eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I am trying to load a bunch of pdf files and query them using OpenAI APIs. from langchain.text_splitter import CharacterTextSplitter #from langchain.document_loaders import UnstructuredFileLoader from langchain.document_loaders import UnstructuredPDFLoader from langchain.vectorstores.faiss import FAISS from langchain.embeddings import OpenAIEmbeddings import pickle import os print("Loading data...") pdf_folder_path = "content/" print(os.listdir(pdf_folder_path)) # Load multiple files # location of the pdf file/files. loaders = [UnstructuredPDFLoader(os.path.join(pdf_folder_path, fn)) for fn in os.listdir(pdf_folder_path)] print(loaders) alldocument = [] vectorstore = None for loader in loaders: print("Loading raw document..." + loader.file_path) raw_documents = loader.load() print("Splitting text...") text_splitter = CharacterTextSplitter( separator="\n\n", chunk_size=800, chunk_overlap=100, length_function=len, ) documents = text_splitter.split_documents(raw_documents) #alldocument = alldocument + documents print("Creating vectorstore...") embeddings = OpenAIEmbeddings() vectorstore = FAISS.from_documents(documents, embeddings) #with open("vectorstore.pkl", "wb") as f: with open("vectorstore.pkl", "ab") as f: pickle.dump(vectorstore, f) f.close()  I am trying to load multiple files for QnA but the index only remembers the last file uploaded from a folder. Do I need to change the structure of for loop or have another parameter with the Open Method?

Answer: The problem is that with each iteration of the loop, you're overwriting the previous vectorstore when you create a new one. Then, when saving to "vectorstore.pkl", you're only saving the last vectorstore. print("Loading data...") pdf_folder_path = "content/" print(os.listdir(pdf_folder_path)) # Load multiple files loaders = [UnstructuredPDFLoader(os.path.join(pdf_folder_path, fn)) for fn in os.listdir(pdf_folder_path)] print(loaders) all_documents = [] for loader in loaders: print("Loading raw document..." + loader.file_path) raw_documents = loader.load() print("Splitting text...") text_splitter = CharacterTextSplitter( separator="\n\n", chunk_size=800, chunk_overlap=100, length_function=len, ) documents = text_splitter.split_documents(raw_documents) all_documents.extend(documents) print("Creating vectorstore...") embeddings = OpenAIEmbeddings() vectorstore = FAISS.from_documents(all_documents, embeddings) with open("vectorstore.pkl", "wb") as f: pickle.dump(vectorstore, f)
Question: I have an environment in Anaconda called myEnv. I am trying to install pyinstaller to it. I tried all 3 of the options given here https://anaconda.org/conda-forge/pyinstaller for installing pyinstaller, however, none of them worked. This is what the message looks like in Anaconda Prompt: Collecting package metadata (current_repodata.json): done Solving environment: failed with initial frozen solve. Retrying with flexible solve. Collecting package metadata (repodata.json): done Solving environment: failed with initial frozen solve. Retrying with flexible solve. PackagesNotFoundError: The following packages are not available from current channels: -pyinstaller  This is weird since I do have conda-forge as a channel: conda config --show channels  The response is: channels: - defaults - conda-forge  As a last resort, I tried pip install pyinstaller and that gave an error. WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProxyError('Cannot connect to proxy.', OSError('Tunnel connection failed: 407 Proxy Authentication Required'))': /simple/pyinstaller/ ERROR: Could not find a version that satisfies the requirement pyinstaller (from versions: none) ERROR: No matching distribution found for pyinstaller WARNING: There was an error checking the latest version of pip.  Edit: I also tried conda update conda but that still didn't seem to do the trick. Installing it from the Anaconda channel also didn't work: conda install -c anaconda pyinstaller  It just gave me the same error when I tried to install pyinstaller from the conda-forge channel. Some Extra Details: My Python version: 3.10.9 Laptop: Windows 64

Answer: You can directly install pyinstaller from GitHub python -m pip install git+https://github.com/pyinstaller/pyinstaller  If you're using a conda env file, you can also configure the package to be installed in the same way. name: myEnv channels: dependencies: - pip: - "--editable=git+https://github.com/pyinstaller/pyinstaller" # other dependencies
Question:  python code --> ",".join("one_two_three".split("_")[0:-1])  I want to take the 0th element and last element of delimited string and create a new string put of it . The code should delimited string of any length Expected output : "one,three"  The above code gives me  'one,two'  Can some one please help me

Answer: [0:-1] is a slice, it returns all the elements from 0 to the 2nd-to-last element. Without the third stride parameter, which is used for getting every Nth element, a slice can only return a contiguous portion of a list, not two separate items. You should select the first and last separately, then concatenate them. items = "one_two_three".split("_") result = f"{items[0]},{items[-1]}"
Question: For a given N, I am trying to find every positive integers a and b such that N = a*b. I start decomposing into prime factors using sympy.ntheory.factorint, it gives me a dict factor -> exponent. I have this code already, but I don't want to get duplicates (a and b play the same role): import itertools from sympy.ntheory import factorint def find_decompositions(n): prime_factors = factorint(n) cut_points = {f: [i for i in range(1+e)] for f, e in prime_factors.items()} cuts = itertools.product(*cut_points.values()) decompositions = [((a := np.prod([f**e for f, e in zip(prime_factors, cut)])), n//a) for cut in cuts] return decompositions  Example: In [235]: find_decompositions(12) Out[235]: [(1, 12), (3, 4), (2, 6), (6, 2), (4, 3), (12, 1)]  What I would like to get instead: Out[235]: [(1, 12), (3, 4), (2, 6)]  I tried to reduce halve the range in cut_points with range extends such as e//2, 1 + e//2, (1+e)//2, 1 + (1+e)//2. None of it ended up working. A simple solution is obviously to compute the same and return: decompositions[:(len(decompositions)+1)//2]  but I am looking for an eventual solution that reduces the number of computations instead.

Answer: You're using module sympy, which already has a divisors function: from sympy import divisors print(divisors(12)) # [1, 2, 3, 4, 6, 12] def find_decompositions(n): divs = divisors(n) half = (len(divs) + 1) // 2 # +1 because perfect squares have odd number of divisors return list(zip(divs[:half], divs[::-1])) print(find_decompositions(12)) # [(1, 12), (2, 6), (3, 4)] print(find_decompositions(100)) # [(1, 100), (2, 50), (4, 25), (5, 20), (10, 10)]
Question: I've been using a Python script to access the SIDRA (IBGE) API and fetch data. It was working perfectly fine, but recently, without any changes on my part, I started encountering an SSL error. Here's the code I've been using: import requests url = 'https://servicodados.ibge.gov.br/api/v3/agregados' response = requests.get(url) print(response.json())  Upon running the code, I get the following error:  SSLError: HTTPSConnectionPool(host='servicodados.ibge.gov.br', port=443): Max retries exceeded with url: /api/v3/agregados (Caused by SSLError(SSLError(1, '[SSL: UNSAFE_LEGACY_RENEGOTIATION_DISABLED] unsafe legacy renegotiation disabled (_ssl.c:1007)')))  Here are the things I've tried:  Using the verify=False option with requests.get(). Updating the requests and urllib3 libraries. Trying other libraries like httpx and aiohttp. Checking the OpenSSL version.  Nothing seems to solve the issue. I'm at a loss since the code used to work without any issues. Has anyone encountered this and knows a fix?

Answer: Try: import ssl import requests class TLSAdapter(requests.adapters.HTTPAdapter): def init_poolmanager(self, *args, **kwargs): ctx = ssl.create_default_context() ctx.set_ciphers("DEFAULT@SECLEVEL=1") ctx.options |= 0x4 # <-- the key part here, OP_LEGACY_SERVER_CONNECT kwargs["ssl_context"] = ctx return super(TLSAdapter, self).init_poolmanager(*args, **kwargs) url = "https://servicodados.ibge.gov.br/api/v3/agregados" with requests.session() as s: s.mount("https://", TLSAdapter()) print(s.get(url).json())  Prints: [{'id': 'D5', 'nome': 'Áreas Urbanizadas', 'agregados': [{'id': '8418', 'nome': 'Áreas urbanizadas, ...
Question: In C# the concept of global using can be used to DRY (do not repeat yourself) many common using statements across many files. Now we have many similar imports across many Python files. Is there a similar strategy that we can use to reduce our boilerplate code?

Answer: You could create a "prelude" module whose only purpose is to be glob imported by other modules. This idea can be found in other programming languages, like std::prelude from Rust and Prelude from Haskell. For example, create a prelude.py file with contents like from .sprockets import LeftSprocket, RightSprocket, make_sprocket from .widgets import FooWidget, BarWidget from .util import sprocket2widget, widget2sprokect  Then from whatever script you want to use it from, import it as from my_package.my_module.prelude import *  All of the names from the prelude (LeftSprocket, FooWidget, etc.) will be immediately available. This approach has some nice advantages:  It's entirely optional to the end users. Users can freely decide between using the prelude vs explicitly importing the modules they need. It's non-invasive to the rest of your package. Providing a prelude doesn't require modifying any other modules or otherwise re-organizing your package.  That said, I wouldn't necessarily recommend creating preludes for all your projects. There are some significant disadvantages that it comes with:  There's now multiple modules that export the same names, so IDEs may have a harder time automatically completing imports. You may end up with your IDE suggesting things like my_package.my_module.prelude import FooWidget instead of my_package.my_module.widget import FooWidget. This can also be confusing to users, who won't know if the FooWidgets from both modules are the same or different without investigating both.  It makes it less obvious where global names are coming from. If you glob-import a prelude, there's no easy way to tell what names are available and what modules they come from. This goes against the Zen of Python, Explicit is better than implicit.  There's a risk of name collisions. Consider for example this recent question. The issue there had to do with the OP including both the imports from PIL import Image and from tkinter import *, where the latter silently re-defined Image to be tkinter.Image.   See also Why is "import *" bad?.
Question: EDIT: I extended the example to show the more complex case that I was talking about before. Thanks for the feedback in the comments. Some more context:  I am mostly interested in this on a theoretical level. I would be happy with an answer like "This is not possible because ...", giving some detailed insights about python internals. For my concrete problem I have a solution, as the library allows to pass a class which then can do what I want. However, if there would be a way to use the simpler interface of just passing the function which gets bound, I would save many lines of code. (For the interested: This question is derived from the sqlalchemy extension associationproxy and the creator function parameter.)  # mylib.py from typing import Callable class C: def __init__(self, foo: Callable): self.foo = foo def __get__(self, instance, owner): # simplified return CConcrete(self.foo) class CConcrete: foo: Callable def __init__(self, foo: Callable): self.foo = foo def bar(self): return self.foo()  # main.py from mylib import C def my_foo(self): return True if self else False class House: window = C(my_foo) my_house = House() print(my_house.window.bar())  This code gives the error  my_foo() missing 1 required positional argument: 'self'  How can I get my_foo be called with self without changing the class C itself? The point is that a class like this exists in a library, so I can't change it. In fact it's even more complicated, as foo gets passed down and the actual object where bar exists and calls self.foo is not C anymore. So the solution can also not include assigning something to c after creation, except it would also work for the more complex case described.

Answer: You can bind the method to the instance manually, since it won't be living in the class __dict__: self.foo = foo.__get__(self)  This is assuming you don't want to monkeypatch the entire class. If you do, assign c.foo = my_foo instead of passing to the instance initializer. You could also conceivably use inheritance: class C(C): def __init__(self): super().__init__(self.foo) def foo(self): return True
Question: Does PyNaCl release the Global Interpreter Lock? Will it be suitable to use multithreading for encryption using PyNaCl's SecretBox? I want to encrypt relatively large amounts of data (~500 MB) using PyNaCl. For this, I divide it into chunks of around 5MB and encrypt them using ThreadPoolExecutor(). Is this ideal? I don't know if PyNaCl releases Python's GIL and whether it will actually encrypt the chunks in parallel resulting in performance gains. Edit: To avoid confusion, let me clear the experimental results. After testing the current implementation a bunch of times, I found out that it was slightly faster than encrypting the entire data directly, and extremely faster than a simple for loop. However, I need hard evidence (reference to documentation, or some kind of test) to prove that the task is indeed running in parallel and GIL is not blocking performance. This is my current implementation using ThreadPoolExecutor() from concurrent.futures import ThreadPoolExecutor from os import urandom from typing import Tuple from nacl import secret from nacl.bindings import sodium_increment from nacl.secret import SecretBox def encrypt_chunk(args: Tuple[bytes, SecretBox, bytes, int]): chunk, box, nonce, macsize = args try: outchunk = box.encrypt(chunk, nonce).ciphertext except Exception as e: err = Exception("Error encrypting chunk") err.__cause__ = e return err if not len(outchunk) == len(chunk) + macsize: return Exception("Error encrypting chunk") return outchunk def encrypt( data: bytes, key: bytes, nonce: bytes, chunksize: int, macsize: int, ): box = SecretBox(key) args = [] total = len(data) i = 0 while i < total: chunk = data[i : i + chunksize] nonce = sodium_increment(nonce) args.append((chunk, box, nonce, macsize,)) i += chunksize executor = ThreadPoolExecutor(max_workers=4) out = executor.map(encrypt_chunk, args) executor.shutdown(wait=True) return out 

Answer: PyNaCl uses the Common Foreign Function Interface (CFFI) to provide bindings to the C library Libsodium. We can see that the SecretBox function is basically a binding for the crypto_secretbox() function of the Libsodium library. As per CFFI documentation:  [2] C function calls are done with the GIL released.  Since most of the functions from PyNaCl are bindings to the Libsodium library using CFFI, they will release the Global Interpreter Lock during the execution of the C function. This should explain the performance improvements from multithreading.
Question: The pandas dataframe: data = pd.DataFrame ({ 'group': ['A', 'A', 'B', 'B', 'C', 'C'], 'date': ['2023-01-15', '2023-02-20', '2023-01-10', '2023-03-05', '2023-02-01', '2023-04-10'], 'value': [10, 15, 5, 25, 8, 12]} )  Trying to get the values of the 'value' column based on the min and max values of 'date' column for each 'group' in an aggregate function: ## the following doesn't work output = ( df .groupby(['group'],as_index=False).agg( ## there are some other additional aggregate functions happening here too. value_at_min = ('value' , lambda x: x.loc[x['date'].idxmin()]) , value_at_max = ('value' , lambda x: x.loc[x['date'].idxmax()]) ))  This doesn't work, even with converting date to datetime (in fact, my original date column is in datetime format). Desired output should be:  group min_date max_date value_at_min value_at_max 0 A 2023-01-15 2023-02-20 10 15 1 B 2023-01-10 2023-03-05 5 25 2 C 2023-02-01 2023-04-10 8 12 

Answer: Sort the dataframe by date then groupby and aggregate with nth to get the rows corresponding to min and max dates g = data.sort_values(['date']).groupby('group') g.nth(0).merge(g.nth(-1), on='group', suffixes=['_min', '_max'])    date_min value_min date_max value_max group A 2023-01-15 10 2023-02-20 15 B 2023-01-10 5 2023-03-05 25 C 2023-02-01 8 2023-04-10 12
Question: I have a function that always returns an empty list (it's a long story why), I could type hint as usual with just "list", but it would be useful to indicate that the list is always going to be the same. My first thought was to use literal like this: from typing import Literal def get_empty_list() -> Literal[[]]: return []  Mypy flags this as an invalid type, is there a correct way to type hint an always empty list? (obviously, I could type hint just a list, but this is less helpful) To be explicit, this is a list that is always empty and doesn't expect to have any elements. (As seperate for example, from a list that is currently empty, but might have elements of some type added later on).

Answer: If you want a type that expresses "list that doesn't have elements now, but might have elements added later", then that's fundamentally not a static type. Appending elements to such a list would be valid, but would cause the list to stop being an element of that type, which should not happen with static types in a program that respects static typing. If you want something that expresses "list that has no elements, and will never have elements", you can kind of do it: from typing import Never def get_empty_list() -> list[Never]: return []  Here, we've annotated the element type as typing.Never, a type with no values. This is actually the type mypy infers for an empty list literal if there's no context to suggest another type, as you can see with reveal_type: from typing import Never x: list[Never] reveal_type(x) reveal_type([])  mypy output: main.py:4: note: Revealed type is "builtins.list[<nothing>]" main.py:5: note: Revealed type is "builtins.list[<nothing>]" Success: no issues found in 1 source file  Here are the implications of using a list[Never] type:  Static type checkers will not allow operations that add elements to a list annotated as list[Never].  They will allow operations like get_empty_list().append(sys.exit()), where the argument to append is an expression that cannot evaluate to a value.  They will recognize that get_empty_list()[0] cannot evaluate to a value... but this is very different from considering it a prohibited operation. In fact, rather than prohibiting the operation, they will allow it, in almost any context. After all, if an expression cannot produce a value, then it cannot produce a value of the wrong type, no matter what the right type is. So you can do stuff like x: int = get_empty_list()[0]  and from a static typing perspective, that's perfectly valid. ([][0] is considered valid too.)  They will not recognize that len(get_empty_list()) must be 0.
Question: The Python module array supports arrays, which are, unlike lists, stored in a contiguous manner, giving performance characteristics which are often more desirable. But the types of elements are limited to those listed in the documentation. I can see why the types have to be ones with constant (or at least bounded from above) representation size. But don't pointers fall into that category? (The main implementation is written in C which, admittedly, allows pointers to different types to have different sizes, but they're all (perhaps except pointers to C functions, but that's not an issue for this question) convertible to intptr_t.) So, given boxing, arrays of arbitrary Python objects could be easily implemented, right? So why aren't they?

Answer: because we have lists, which are exactly an array of pointers. python's array module stores data contiguously, python objects have dynamic sizes and cannot be stored contiguously, so we have lists which store the pointers. the main use of array is for passing around C-style arrays between different C functions, arrays are slower than lists for numeric types due to the boxing and unboxing, so it has no use in any pure python code.
Question: In the source code of Stable Baselines3, in common/preprocessing.py, on line 158, there is: return (int(len(observation_space.nvec)),). As far as I know, in Python, len can only return an int, and regardless if this is not true, would return an int here (might be wrong on both counts). If this is the case, the cast to int would not make sense to me. Am I missing something here?

Answer: The call to int is not necessary. len always returns an int. If a class' __len__ implemention returns something other than an int, Python will attempt to convert it to an int. If that can't be done, a TypeError will be raised. There's no scenario where len can return something other than an int. You can try to violate this yourself: class Foo: def __len__(self): return "bad" >>> len(Foo()) Traceback (most recent call last) ... TypeError: 'str' object cannot be interpreted as an integer
Question: I am trying to install (through pyenv) Python-3.11.4 under CentOS-7. It installs but without GUI. I get the following error message: Installing Python-3.11.4... Traceback (most recent call last): File "<string>", line 1, in <module> File "/.../pyenv/versions/3.11.4/lib/python3.11/tkinter/__init__.py", line 38, in <module> import _tkinter # If this fails your Python may not be configured for Tk ^^^^^^^^^^^^^^^ ModuleNotFoundError: No module named '_tkinter' WARNING: The Python tkinter extension was not compiled and GUI subsystem has been detected. Missing the Tk toolkit? Installed Python-3.11.4 to /.../pyenv/versions/3.11.4  While Python-3.9.16 installs successfully on the same machine. According to Python 3.11 Build Changes, the requirement is to have "Tcl/Tk version 8.5.12 or newer" installed. I have $ rpm -q tk tk-devel tcl tcl-devel tk-8.5.13-6.el7.x86_64 tk-devel-8.5.13-6.el7.x86_64 tcl-8.5.13-8.el7.x86_64 tcl-devel-8.5.13-8.el7.x86_64  The same page says "Tcl/Tk, and uuid flags are detected by pkg-config (when available). tkinter now requires a pkg-config command to detect development settings for Tcl/Tk headers and libraries.", which is also installed: $ rpm -q pkgconfig pkgconfig-0.27.1-4.el7.x86_64  Could you please help me to understand what might be the reason of the failure to install _tkinter? Thank you very much for your help!

Answer: To build successfully Python-3.11.4 under CentOS-7, one needs to set the following environment variables: export CPPFLAGS="$(pkg-config --cflags openssl11) -I/usr/include" export LDFLAGS="$(pkg-config --libs openssl11) -L/usr/lib64 -ltcl8.5 -ltk8.5"  where the openssl11 parts are needed for the _ssl module, and the rest is needed for the _tkinter module. The pieces needed to build Python with the Tk/Tcl were found in  /usr/lib64/tclConfig.sh /usr/lib64/tkConfig.sh  To check if tkinter was built successfully, do the following: /path/to/python/3.11.4/bin/python3 -m tkinter  It seems, the root cause of the problem is that tcl and tk rpm packages in CentOS-7 do not provide the corresponding pkg-config files:  /usr/lib64/pkgconfig/tcl.pc /usr/lib64/pkgconfig/tk.pc  and so one has to provide the corresponding information manually.