eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I'm a newbie with Python and been trying to install modules using pip unsuccessfully in my small project. Following advice online, I've created my own virtual environment and imported my first module cowsay fine. I can definitely see the module being installed in my project:  BUT, when attempting to run the file in my terminal, I keep getting a ModuleNotFoundError. (env) sr@python-virtual-env >> pip install cowsay Collecting cowsay Using cached cowsay-5.0-py2.py3-none-any.whl Installing collected packages: cowsay Successfully installed cowsay-5.0 (env) sr@python-virtual-env >> python say.py John Traceback (most recent call last): File "/Users/sr/Sites/python-virtual-env/say.py", line 1, in <module> import cowsay ModuleNotFoundError: No module named 'cowsay'  What am I missing here? Thanks in advance!

Answer: As far as the output of the command which python pip is python: aliased to /usr/bin/python3 /Users/sr/Sites/python-virtual-env/env/bin/pip  we can say that you are running the python not from your environment. So it cannot see by default any package installed in this env. Remove the alias python (in bash, the command unalias python should be enough for the current session) or print the command with the leading \ to ignore aliases and look for the python in $PATH, like this $ \python say.py John  Note \ at the beginning of this command. But if you have declared python as a shell function, this function may be executed (hardly so, but what if some shell behaves this way?). So may be the best way to resolve the problem is to run env python, like this: $ env python say.py John  Important update Having thought somewhat longer about this problem, I can allow myself a categorical answer: env python ... is the only way to run python directly from PATH. Here's the model to see why I think so. Let's say we run bash on a linux machine with PATH='/usr/bin:/bin' and the only python /usr/bin/python: $ bash --version | head -1 GNU bash, version 4.4.20(1)-release (x86_64-pc-linux-gnu) $ export PATH='/usr/bin:/bin' $ which -a python /usr/bin/python  Making sure we don't have any aliases or functions named python: $ type -a python python is /usr/bin/python  Now we define an alias and a function with name python: $ alias python='echo Running python as an alias; /usr/bin/python' $ function python(){ echo Running python as a function; python "$@"; }  Let's see what is the answer of which and type commands now: $ which -a python /usr/bin/python $ type -a python python is aliased to `echo Running python as an alias; /usr/bin/python' python is a function python () { echo Running python as a function; echo Running python as an alias; /usr/bin/python "$@" } python is /usr/bin/python  And here we go: $ python -c 'print("Hello world")' Running python as an alias Hello world $ \python -c 'print("Hello world")' Running python as a function Running python as an alias Hello world $ env python -c 'print("Hello world")' Hello world  As we can see, an alias took precedence over a function in this case, and only env python helped to reach the PATH directly, which is most desirable when running python from a local environment.
Question: I have a py_binary executable target, and want to use this from a custom rule. I am able to get this to work, but only by duplicating the dependencies of my py_binary target with my custom rule. Is there any way to avoid this duplication and automatically include the dependencies of the py_binary? I have simplified my problem down to the following (reproduce by running the //foo:main target) greet/BUILD.bazel: py_binary( name = "greet", srcs = ["greet.py"], deps = ["@rules_python//python/runfiles"], visibility = ["//visibility:public"], )  greet/greet.py: from rules_python.python.runfiles import runfiles print("hello world")  foo/BUILD.bazel: load("//bazel:defs.bzl", "foo_binary") foo_binary( name = "main", )  bazel/BUILD.bazel: (empty) bazel/defs.bzl: def _foo_binary(ctx): shell_script = ctx.actions.declare_file("run.sh") ctx.actions.write( output = shell_script, content = """#!/bin/bash $0.runfiles/svkj/{runtime} """.format(runtime=ctx.executable._greet.short_path), is_executable = True, ) return [ DefaultInfo( executable = shell_script, runfiles = ctx.runfiles( files = ctx.files._greet + ctx.files.deps + ctx.files._python, collect_data = True ), ), ] foo_binary = rule( implementation = _foo_binary, attrs = { "deps": attr.label_list( allow_files=True, ), "_greet": attr.label( default = "//greet:greet", cfg = "exec", executable = True, ), # TODO - Why is it necessary to add this explicitly? "_python": attr.label_list( allow_files=True, default = ["@python3_9//:python3", "@rules_python//python/runfiles"], ), }, executable = True, )  bazel/BUILD.bazel:  WORKSPACE: workspace(name = "svkj") load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_archive") http_archive( name = "rules_python", sha256 = "5fa3c738d33acca3b97622a13a741129f67ef43f5fdfcec63b29374cc0574c29", strip_prefix = "rules_python-0.9.0", url = "https://github.com/bazelbuild/rules_python/archive/refs/tags/0.9.0.tar.gz", ) load("@rules_python//python:repositories.bzl", "python_register_toolchains") python_register_toolchains( name = "python3_9", python_version = "3.9", ) load("@python3_9//:defs.bzl", "interpreter") 

Answer: You need to correctly collect the runfiles of the _greet binary. Instead of just using the _greet binary itself, try the following to include its dependencies (and data dependencies) aswell: DefaultInfo( executable = shell_script, runfiles = ctx._greet[DefaultInfo].default_runfiles )  See also: https://bazel.build/extending/rules#runfiles
Question: I have a data frame column as float64 full of NaN values, If I cast it again to float64 they got substituted for <NA> values which are not the same. I know that the <NA> values are pd.NA, while NaN values are np.nan , so they are different things. So why casting an already float64 column to float64 changed NaN to <Na> ? Here's an example: df=pd.DataFrame({'a':[1.0,2.0]}) print(df.dtypes) #output is: float64 df['a'] = np.nan print(df.dtypes) # output is float64 print(df) a 0 NaN 1 NaN #Now, lets cast that float64 to float 64 df3['a']=df3['a'].astype(pd.Float64DType()) print(df3.dtypes) #output is Float64, notice it's uppercase F this time, previously it was lowercase print(df3) a 0 <NA> 1 <NA>  it seems float64 and Float64 are two different things. And NaN (np.nan) is the null value for float64 while <NA> (pd.NA) is the null for Float64 Is this correct? And if so, what's under the hoods?

Answer: Yes, you are correct. float64 and Float64 are two different data types in pandas. The difference is that Float64 is an extension type that can hold missing values using a special  sentinel, while float64 is a native numpy type that uses NaN to represent missing values. Under the hood, Float64 uses a numpy array with dtype object to store the values, while float64 uses a numpy array with dtype float64. This means that Float64 may have some performance overhead compared to float64, but it also allows more consistent handling of missing values across different data types. Check this out: Numpy float64 vs Python float
Question: I have the following directory for a project: photo_analyzer/ │ ├── main.py # Main script to run the photo analyzer ├── gui/ # Directory for GUI-related files │ ├── __init__.py # Package initialization │ ├── app.py # Tkinter application class and main GUI logic │ ├── widgets.py # Custom Tkinter widgets (if needed) │ ├── styles.py # Styling configurations for the GUI │ └── resources/ # Directory for GUI resources (e.g., icons, images) │ ├── icon.png # Application icon │ └── ... │ ├── photo_analysis/ # Directory for photo analysis-related files │ ├── __init__.py # Package initialization │ ├── photo_utils.py # Utility functions for photo manipulation │ ├── light_profiles.py # Functions to analyze light profiles in photos │ └── ...  I'm trying to import photo_utils and light_profiles from the photo_analysis module into gui/app.py. Whenever I try to run app.py, I get the following error message: Traceback (most recent call last): File "c:\Users\Josh\Documents\PhotoAnalyzer\gui\app.py", line 6, in <module> from photo_analysis import photo_utils ModuleNotFoundError: No module named 'photo_analysis'  I have empty __init__.py files in both directories, so I don't think that's the issue. I've tried adding the photo_analysis module to the Python path using SET PYTHONPATH="C:\Users\Josh\Documents\PhotoAnalyzer\photo_analysis" and I've tried to use the sys.path technique as well. Both end with the same error message as before. photo_utils.py imports widgets.py from the "gui" module. Could I be experiencing circular dependency? I did try the work-around explained in text by importing widgets into each method in photo_utils that used it, and while this allowed a test main code in photo_utils to run properly, app.py still gives the same error message that there is no module named 'photo_analysis'. What am I missing?

Answer: It is a tricky one. Normally python imports from their sibling files or childs of sibling folders. Here if you want to import from a file that are from same parent then you have to use sys.path.append('.') Secondly it also depends on the terminal current working directory. If your current working directory is "photo_analyzer" then adding these line will work import sys sys.path.append('.') import photo_analysis.photo_utils as pa  Otherwise if your current working directory is "photo_analyzer/gui", then you have to add these lines import sys sys.path.append('./../photo_analysis') import photo_utils as pa
Question: Context: I am trying to query Llama-2 7B, taken from HuggingFace (meta-llama/Llama-2-7b-hf). I give it a question and context (I would guess anywhere from 200-1000 tokens), and ask it to answer the question based on the context (context is retrieved from a vectorstore using similarity search). Here are my two problems:  The answer ends, and the rest of the tokens until it reaches max_new_tokens are all newlines. Or it just doesn't generate any text and the entire response is newlines. Adding a repetition_penalty of 1.1 or greater has solved infinite newline generation, but does not get me full answers. For answers that do generate, they are copied word for word from the given context. This remains the same with repetition_penalty=1.1, and making the repetition penalty too high makes the answer nonsense.  I have only tried using temperature=0.4 and temperature=0.8, but from what I have done, tuning temperature and repetition_penalty both result in either the context being copied or a nonsensical answer. Note about the "context": I am using a document stored in a Chroma vector store, and similarity search retrieves the relevant information before I pass it to Llama. Example Problem: My query is to summarize a certain Topic X. query = "Summarize Topic X"  The retrieved context from the vectorstore has 3 sources that looks something like this (I format the sources in my query to the LLM separated by newlines): context = """When talking about Topic X, Scenario Y is always referred to. This is due to the relation of Topic X is a broad topic which covers many aspects of life. No one knows when Topic X became a thing, its origin is unknown even to this day."""  Then the response from Llama-2 directly mirrors one piece of context, and includes no information from the others. Furthermore, it produces many newlines after the answer. If the answer is 100 tokens, and max_new_tokens is 150, I have 50 newlines. response = "When talking about Topic X, Scenario Y is always referred to. This is due to the relation of \n\n\n\n"  One of my biggest issues is that in addition to copying one piece of context, if the context ends mid-sentence, so does the LLM response. Is anyone else experiencing anything like this (newline issue or copying part of your input prompt)? Has anyone found a solution?

Answer: This is a common issue with pre-trained base models like Llama. My first thought would be to select a model that has some sort of instruction tuning done to it i.e https://huggingface.co/meta-llama/Llama-2-7b-chat. Instruction tuning impacts the model's ability to solve tasks reliably, as opposed to the base model, which is often just trained to predict the next token (which is often why the cutoff happens). The second thing, in my experience, I have seen that has helped is using the same prompt format that was used during training. You can see in the source code the prompt format used in training and generation by Meta. Here is a thread about it. Finally, for repetition, using a Logits Processor at generation-time has been helpful to reduce repetition.
Question: I'm trying to use Series.str.rsplit to return everything up to the delimiter, but can't seem to get past this TypeError I'm receiving: TypeError: StringMethods.rsplit() takes from 1 to 2 positional arguments but 3 were given  I am using pandas 2.0.3. Here is my code: import pandas as pd df1 = pd.DataFrame({'name': ['tom ent', 'nick', 'krish std', 'jack whatever'], 'age': ['5', '6', '7', '8']}) df1['name'] = df1['name'].str.rsplit(' ', 1).str[0]  Output should look like: name age tom 5 nick 6 krish 7 jack 8  Not sure what I'm missing here.

Answer: In Pandas 1.4 and before, the .rsplit method used ordinary parameters, so a call like this would mean that ' ' is the delimiter and 1 is the maximum number of times to split - just like the built-in method of the str class. However, starting in 1.5, n and expand are keyword-only parameters. To use them, the name needs to be specified explicitly, like: .rsplit(' ', n=1). The idea is that, because the programmer is forced to use names, there is no chance of mistaking the order.
Question: I am trying to simulate an option pricing scheme by defining functions (Black-Scholes and Crank-Nicolson) and comparing them. The implementation of the Crank-Nicolson scheme with boundary conditions is as follows (I include the full code for replication): import numpy as np from scipy.stats import norm from scipy.integrate import solve_ivp import matplotlib.pyplot as plt def bs_call(S, K, r, sigma, T): d1 = (np.log(S/K) + (r + 0.5*sigma**2)*T) / (sigma*np.sqrt(T)) d2 = d1 - sigma*np.sqrt(T) return S*norm.cdf(d1) - K*np.exp(-r*T)*norm.cdf(d2) def crank_nicolson(S0, K, r, sigma, T, Smin, Smax, M, N): # Initialization dt = T/N ds = (Smax - Smin)/M S = np.linspace(Smin, Smax, M+1) tau = np.linspace(0, T, N+1) V = np.zeros((M+1, N+1)) # Boundary conditions V[:, 0] = np.maximum(S-K, 0) V[0, :] = 0 V[-1, :] = Smax-K*np.exp(-r*tau) # Tridiagonal matrix for the implicit step alpha = 0.25*dt*(sigma**2*S**2/ds**2 - r*S/ds) beta = -dt*0.5*(sigma**2*S**2/ds**2 + r) gamma = 0.25*dt*(sigma**2*S**2/ds**2 + r*S/ds) A = np.diag(alpha[2:], -1) + np.diag(1+beta[1:]) + np.diag(gamma[:-2], 1) B = np.diag(-alpha[2:], -1) + np.diag(1-beta[1:]) + np.diag(-gamma[:-2], 1) # Time loop for n in range(1, N+1): b = np.dot(B, V[1:-1, n-1]) b[0] -= alpha[1]*V[0, n] b[-1] -= gamma[-2]*V[-1, n] V[1:-1, n] = np.linalg.solve(A, b) # Interpolation for S0 i = int(round((S0-Smin)/ds)) if i == M+1: i = M elif i == 0: i = 1 a = (V[i+1, -1]-V[i, -1])/ds b = (V[i+1, -1]-2*V[i, -1]+V[i-1, -1])/ds**2 return V[i, -1] + a*(S0-S[i]) + 0.5*b*(S0-S[i])**2  Up to here, so far so good. Afterwards, I run an example as follows (full code also included for replication): # Example usage S0 = 100 K = 100 r = 0.05 sigma = 0.2 T = 1 Smin = 0 Smax = 200 M = 200 N = 1000 option_price = crank_nicolson(S0, K, r, sigma, T, Smin, Smax, M, N)  However, I got an error here with the following message: ValueError: shapes (200,200) and (199,) not aligned: 200 (dim 1) != 199 (dim 0)  and I do not know if it comes from the function definition (first part) or from the parameters definition (second script) and how can be solved.

Answer: The error is caused since B.shape[1] is M, whereas V[1:-1, n-1].shape[0] is M-1, which do not match, thus I'd guess the error is due to the function definition. Changing A = np.diag(alpha[2:], -1) + np.diag(1+beta[1:]) + np.diag(gamma[:-2], 1) B = np.diag(-alpha[2:], -1) + np.diag(1-beta[1:]) + np.diag(-gamma[:-2], 1)  to A = np.diag(-alpha[2:M], -1) + np.diag(1-beta[1:M]) + np.diag(-gamma[1:M-1], 1) B = np.diag(alpha[2:M], -1) + np.diag(1+beta[1:M]) + np.diag(gamma[1:M-1], 1)  (heavily "inspired" by this very similar implementation) gets rid of the error message and running cn_price = crank_nicolson(S0, K, r, sigma, T, Smin, Smax, M, N) bs_price = bs_call(S0, K, r, sigma, T) print(cn_price, bs_price)  (with otherwise unchanged code) prints 10.3201002605872 10.450583572185565  Note that I am not familiar with the Crank-Nicolson method and can therefore only hope that the proximity of those values indicates a fix (i.e. correct adjustment) indeed, as opposed to a mere ridding of the error message.
Question: Apologies for the perhaps confusing title. Suppose you have a range (0,10000) and a list of smaller sub-ranges that may overlap [(20,3460),(200,3000),(2800,8704)] and you want to return the ranges of the original range NOT covered by the sub-ranges, in this case [(0, 19), (8705, 10000)] I am trying to figure out how to do this efficiently to work with larger ranges and more numerous sub-ranges. I thought about using itertools.chain to create a single range from the list of sub-ranges to take care of overlaps and then somehow using that range to slice the larger range, but I'm not sure if this is the best approach. If it helps to know the purpose, I am trying to create a list of coding sequences that are wholly or partially in the unaligned regions of a genomic comparison given the ranges of the aligned regions and ranges of the coding sequences. If the coding sequence is only partially aligned, I want to be able to return the ranges that are unaligned.

Answer: intspan can be used to get the complement of the sub ranges. from intspan import intspan start, end = (0,10000) subs = [(20,3460),(200,3000),(2800,8704)] span = intspan.from_ranges(subs) comp = span.complement(low=start, high=end).ranges() print(comp)  Prints: [(0, 19), (8705, 10000)]
Question: I'm trying to port some code from MATLAB to PYTHON and I realized gevfit function in MATLAB seems to behave differently from scipy genextreme, so I realized this minimal example: MATLAB % Create the MATLAB array x = [0.5700, 0.8621, 0.9124, 0.6730, 0.5524, 0.7608, 0.2150, 0.5787, ... 0.7210, 0.7826, 0.8181, 0.5449, 0.7501, 1.1301, 0.7784, 0.5378, ... 0.9550, 0.9623, 0.6865, 0.6863, 0.6153, 0.4372, 0.5485, 0.6318, ... 0.5501, 0.8333, 0.8044, 0.9111, 0.8560, 0.6178, 1.0688, 0.7535, ... 0.7554, 0.7123, 0.7589, 0.8415, 0.7586, 0.3865, 0.3087, 0.7067]; disp(numbers); parmHat = gevfit(x); disp('Estimated parameters (A, B):'); disp(parmHat);   Estimated parameters (A, B): -0.3351 0.1962 0.6466  PYTHON import numpy as np import scipy.stats as stats x = np.array([0.5700, 0.8621, 0.9124, 0.6730, 0.5524, 0.7608, 0.2150, 0.5787, 0.7210, 0.7826, 0.8181, 0.5449, 0.7501, 1.1301, 0.7784, 0.5378, 0.9550, 0.9623, 0.6865, 0.6863, 0.6153, 0.4372, 0.5485, 0.6318, 0.5501, 0.8333, 0.8044, 0.9111, 0.8560, 0.6178, 1.0688, 0.7535, 0.7554, 0.7123, 0.7589, 0.8415, 0.7586, 0.3865, 0.3087, 0.7067]) # Fit the GEV distribution to the data parameters3 = stats.genextreme.fit(x) print("Estimated GEV parameters:", parameters3)   Estimated GEV parameters: (1.0872284332032054, 0.534605335200113, 0.6474387313912493)  I'd expect the same parameters, but results are totally different. Any help?

Answer: The method genextreme.fit is failing to compute the correct result. You can help it generate the correct value by providing initial values for the numerical solver that are better than the default used by genextreme.fit. The initial values are set by providing values for the shape, location and scale parameters: In [29]: from scipy.stats import genextreme In [30]: x = np.array([0.5700, 0.8621, 0.9124, 0.6730, 0.5524, 0.7608, 0.2150, 0.5787, ...: 0.7210, 0.7826, 0.8181, 0.5449, 0.7501, 1.1301, 0.7784, 0.5378, ...: 0.9550, 0.9623, 0.6865, 0.6863, 0.6153, 0.4372, 0.5485, 0.6318, ...: 0.5501, 0.8333, 0.8044, 0.9111, 0.8560, 0.6178, 1.0688, 0.7535, ...: 0.7554, 0.7123, 0.7589, 0.8415, 0.7586, 0.3865, 0.3087, 0.7067]) ...: In [31]: genextreme.fit(x, 0.34, loc=0.65, scale=0.20) # Include initial guess of the parameters Out[31]: (0.33513328610099824, 0.6466250071208526, 0.19615018966970216)  Note that the parameter c used by SciPy's genextreme is the negative of the parameter k in Matlab. Also note that the order of the parameters in SciPy is c, location, scale, while in Matlab it is k, scale, location.
Question: I have the following Pandas DataFrame Key Value A 10 A 20 B 30 B 40 C 50 A 60 A 70 A 70 B 80 A 90  And I need to create an index that auto increment only when the key repeats after sequence of different keys. So, I need this output: Key Value Index A 10 1 A 20 1 B 30 1 B 40 1 C 50 1 A 60 2 A 70 2 A 70 2 B 80 2 A 90 3  Thanks! I try with the method groupby and cumcount() + 1 but it does not work.

Answer: Using an ordered Categorical and numpy.cumsum: import numpy as np s = pd.Categorical(df['Key'], ordered=True) df['Index'] = np.cumsum(s<s.shift())+1  If you want a custom order pass categories=['X', 'Z', 'Y']. Or, like @SimonT commented, if your categories are lexicographically sorted: df['Index'] = np.cumsum(df['Key']<df['Key'].shift())+1  Output:  Key Value Index 0 A 10 1 1 A 20 1 2 B 30 1 3 B 40 1 4 C 50 1 5 A 60 2 6 A 70 2 7 A 70 2 8 B 80 2 9 A 90 3
Question: I am trying to use list comprehension for below code # Input values = [1, 2, 3, 4]  # Using for loop ans = [] for value in values: if value == 1: ans.append(value + value) elif value == 4: ans.append(value * value) print(f"Ans: {ans}")  # Using list comprehension ans = [ (value + value) if value == 1 else (value * value) if value == 4 for value in values ] print(f"Ans: {ans}")  But getting error for list comprehension as below:  SyntaxError: expected 'else' after 'if' expression  I tried defining else part like below, but it did not work: else pass else continue  The output should be [2, 16]. Other values are skipped. What I am missing?

Answer: I'm guessing that this is a small example of something more complicated, but as it is it is fairly easy to create a comprehension like: values = [1, 2, 3, 4] ans = [ value + value if value == 1 else value * value for value in values if value in [1,4] ] print(ans)
Question: I would like to remove the string from a string column when it contains a lower letter (the string column may be NaN or include multiple string in one row)     Column1 Column2 Column3     NaN NaN NaN   BCDE8ENGUGUETNJN NaN NaN   vk4JGFIEh5 NaN NaN   t7chJGWYEj BCSTACK BCTENSORFLOW   5hNIE7EEP OVERFLOW NaN     The original df is look like the above I have tried the "str.contains" function to locate and replace it when it contains lowerletter Since str function could not be used for NaN value, I replaced the NaN value with string 'nan' first, then aimed to replace all the lower letter with regex. Since 'nan' is also a lower letter, it should be replaced as well  df['Column1'].fillna('nan',inplace=True) df['Column2'].fillna('nan',inplace=True) df['Column3'].fillna('nan',inplace=True) lowerletterpattern = r'[a-z]*' mask1 = df['Column1'].str.contains(lowerletterpattern) df.loc[mask1,'Column1'] = np.nan mask2 = df['Column2'].str.contains(lowerletterpattern) df.loc[mask2,'Column2'] = np.nan mask3 = df['Column3'].str.contains(lowerletterpattern) df.loc[mask3,'Column3'] = np.nan   but the df returned with all NaN value Below is the expected result:     Column1 Column2 Column3     NaN NaN NaN   BCDE8ENGUGUETNJN NaN NaN   NaN NaN NaN   NaN BCSTACK BCTENSORFLOW   NaN OVERFLOW NaN    

Answer: One option, check for [a-z] with str.contains and mask: out = df.mask(df.apply(lambda s: s.str.contains('[a-z]').fillna(True)))  Or, with replace: out = df.replace('.*[a-z].*', float('nan'), regex=True)  Output:  Column1 Column2 Column3 0 NaN NaN NaN 1 BCDE8ENGUGUETNJN NaN NaN 2 NaN NaN NaN 3 NaN BCSTACK BCTENSORFLOW 4 NaN OVERFLOW NaN