eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: Suppose I have a dataframe like this import pandas as pd df = pd.DataFrame({'ID':[1,1,1,1,1,2,2,2,2], 'Order':[1,2,3,4,5,1,2,3,4]}) df ID Order 0 1 1 1 1 2 2 1 3 3 1 4 4 1 5 5 2 1 6 2 2 7 2 3 8 2 4  I need to upsample by entire blocks, so essentially creating new copies of the block for ID == 1, and ID == 2. So the upsampled df (for n = 2 two samples with replacement) might look something like  df_upsampled = pd.DataFrame({'ID':[1,1,1,1,1,2,2,2,2,1,1,1,1,1,2,2,2,2], 'Order':[1,2,3,4,5,1,2,3,4,1,2,3,4,5,1,2,3,4]}) df_upsampled ID Order 0 1 1 1 1 2 2 1 3 3 1 4 4 1 5 5 2 1 6 2 2 7 2 3 8 2 4 9 1 1 10 1 2 11 1 3 12 1 4 13 1 5 14 2 1 15 2 2 16 2 3 17 2 4  I thought I could handle this quick with resample() but haven't figured out how to copy entire blocks (per ID)

Answer: So basically all you need to do is to get a copy of your data frame and concat it to the original one: df = pd.concat([df, df], ignore_index=True)
Question: IPython has this new feature that reformats my prompt. Unfortunately, it is really buggy, so I want to disable it. I managed to do it when starting IPython from the command line by adding the following line in my ipython_config.py: c.TerminalInteractiveShell.autoformatter = None  However, it does not work when I run it from a python script. I start IPython from my script the following way: c = traitlets.config.get_config() c.InteractiveShellEmbed.colors = "Linux" c.TerminalInteractiveShell.autoformatter = None c.InteractiveShellEmbed.loop_runner = lambda coro: loop.run_until_complete(coro) IPython.embed(display_banner='', using='asyncio', config=c)  If I change the colors value, the colors change accordingly, so the configuration itself works. However, no matter what I do with autoformatter, IPython autoformats my code regardless. What am I doing wrong?

Answer: Apparently, the answer is: c.InteractiveShellEmbed.autoformatter = None
Question: I want to know which elements of list_1 are in list_2. I need the output as an ordered list of booleans. But I want to avoid for loops, because both lists have over 2 million elements. This is what I have and it works, but it's too slow: list_1 = [0,0,1,2,0,0] list_2 = [1,2,3,4,5,6] booleans = [] for i in list_1: booleans.append(i in list_2) # booleans = [False, False, True, True, False, False]  I could split the list and use multithreading, but I would prefer a simpler solution if possible. I know some functions like sum() use vector operations. I am looking for something similar. How can I make my code more efficient?

Answer: If you want to use a vector approach you can also use Numpy isin. It's not the fastest method, as demonstrated by oda's excellent post, but it's definitely an alternative to consider. import numpy as np list_1 = [0,0,1,2,0,0] list_2 = [1,2,3,4,5,6] a1 = np.array(list_1) a2 = np.array(list_2) np.isin(a1, a2) # array([False, False, True, True, False, False])
Question: I am experimenting with the typing module and I wanted to know how to properly type something like a Nonagon (a 9 point polygon), which should be a Tuple and not a List because it should be immutable. In 2D space, it would be something like this: Point2D = Tuple[float, float] Nonagon = Tuple[Point2D, Point2D, Point2D, Point2D, Point2D, Point2D, Point2D, Point2D, Point2D] nine_points: Nonagon = ( (0.0, 0.0), (6.0, 0.0), (6.0, 2.0), (2.0, 2.0), (6.0, 5.0), (2.0, 8.0), (6.0, 8.0), (6.0, 10.0), (0.0, 10.0), )  Is there any syntactic sugar available to make the Nonagon declaration shorter or easier to read? This is not valid Python, but I am looking for something similar to this: Nonagon = Tuple[*([Point2D] * 9)] # Not valid Python  Or using NamedTuple # Not properly detected by static type analysers Nonagon = NamedTuple('Nonagon', [(f"point_{i}", Point2D) for i in range(9)])  This is NOT what I want: # Valid but allows for more and less than 9 points Nonagon = Tuple[Point2D, ...]  I think the most adequate way would be something like: from typing import Annotated # Valid but needs MinMaxLen and checking logic to be defined from scratch Nonagon = Annotated[Point2D, MinMaxLen(9, 9)] 

Answer: You can use the types module. All type hints come from types.GenericAlias. From the doc:  Represent a PEP 585 generic type E.g. for t = list[int], t.__origin__ is list and t.__args__ is (int,).  This means that you can make your own type hinting by passing the type arguments to the class itself. >>> Point2D = tuple[float, float] >>> Nonagon = types.GenericAlias(tuple, (Point2D,)*9) >>> Nonagon tuple[tuple[float, float], tuple[float, float], tuple[float, float], tuple[float, float], tuple[float, float], tuple[float, float], tuple[float, float], tuple[float, float], tuple[float, float]]
Question: The language interpreter is set to a Python 3.9 version:  But a Python scratch file is being parsed by some kind of 3.6 interpreter:  Note that I created in two different scratch files and the same error occurs. Why would this happen and is there a workaround [short of creating an entirely new project from scratch]? I am on Pycharm Professional 2021.3.1 Update based on answer by @TurePaisson he though maybe the Code is compatible with specific Python were set. That was a shrewd guess - but turns out I have not set that:   Update The following snippet can be used to test python3.6 vs 3.8+ x = (y := 3) + 7 

Answer: Following up on @bad_coder 's attempt to fix that will be paraphrased as:  check the Run Configuration for pointing to a different python interpreter than the project level one  That fix worked for me:  Bring up the Run [Context menu] | Edit Configurations    Change the Python interpreter to the appropriate one: Shown below is a case where the interpreter is said to the earlier language level. Go to the dropdown and select a correct [python 3.8+] interpeter.
Question: So, I was writing an event emitter class using Python. Code currently looks like this: from typing import Callable, Generic, ParamSpec P = ParamSpec('P') class Event(Generic[P]): def __init__(self): ... def addHandler(self, action : Callable[P, None]): ... def removeHandler(self, action : Callable[P, None]): ... def fire(self, *args : P.args, **kwargs : P.kwargs): ...  As you can see, annotations depend on ParamSpec, which was added to typing in python 3.10 only. And while it works good in Python 3.10 (on my machine), it fails in Python 3.9 and older (on other machines) because ParamSpec is a new feature. So, how could I avoid importing ParamSpec when running program or use some fallback alternative, while not confusing typing in editor (pyright)?

Answer: I don't know if there was any reason to reinvent the wheel, but typing_extensions module is maintained by python core team, supports python3.7 and later and is used exactly for this purpose. You can just check python version and choose proper import source: import sys if sys.version_info < (3, 10): from typing_extensions import ParamSpec else: from typing import ParamSpec
Question: Say I have the following dataframe:  values 0 4 1 0 2 2 3 3 4 0 5 8 6 5 7 1 8 0 9 4 10 7  I want to find a pandas vectorized function (preferably using groupby) that would replace all nonzero values with the first nonzero value in that chunk of nonzero values, i.e. something that would give me  values new 0 4 4 1 0 0 2 2 2 3 3 2 4 0 0 5 8 8 6 5 8 7 1 8 8 0 0 9 4 4 10 7 4  Is there a good way of achieving this?

Answer: Make a boolean mask to select the rows having zero and its following row, then use this boolean mask with where to replace remaining values with NaN, then use forward fill to propagate the values in forward direction. m = df['values'].eq(0) df['new'] = df['values'].where(m | m.shift()).ffill().fillna(df['values'])  Result print(df) values new 0 4 4.0 1 0 0.0 2 2 2.0 3 3 2.0 4 0 0.0 5 8 8.0 6 5 8.0 7 1 8.0 8 0 0.0 9 4 4.0 10 7 4.0
Question: Let's say I have a list[str] object containing timestamps in "HH:mm" format, e.g. timestamps = ["22:58", "03:11", "12:21"]  I want to convert it to a list[int] object with the "number of minutes since midnight" values for each timestamp: converted = [22*60+58, 3*60+11, 12*60+21]  ... but I want to do it in style and use a single list comprehension to do it. A (syntactically incorrect) implementation that I naively constructed was something like: def timestamps_to_minutes(timestamps: list[str]) -> list[int]: return [int(hh) * 60 + int(mm) for ts in timestamps for hh, mm = ts.split(":")]  ... but this doesn't work because for hh, mm = ts.split(":") is not a valid syntax. What would be the valid way of writing the same thing? To clarify: I can see a formally satisfying solution in the form of: def timestamps_to_minutes(timestamps: list[str]) -> list[int]: return [int(ts.split(":")[0]) * 60 + int(ts.split(":")[1]) for ts in timestamps]  ... but this is highly inefficient and I don't want to split the string twice.

Answer: You could use an inner generator expression to do the splitting: [int(hh)*60 + int(mm) for hh, mm in (ts.split(':') for ts in timestamps)]   Although personally, I'd rather use a helper function instead: def timestamp_to_minutes(timestamp: str) -> int: hh, mm = timestamp.split(":") return int(hh)*60 + int(mm) [timestamp_to_minutes(ts) for ts in timestamps] # Alternative list(map(timestamp_to_minutes, timestamps))
Question: I have a Dataframe like this:     name phase value     BOB 1 .9   BOB 2 .05   BOB 3 .05   JOHN 2 .45   JOHN 3 .45   JOHN 4 .05   FRANK 1 .4   FRANK 3 .6     I want to find which entry in column 'phase' has the maximum value in column 'value'. If more than one share the same maximum value keep the first or a random value for 'phase'.  Desired result table:     name phase value     BOB 1 .9   JOHN 2 .45   FRANK 3 .6     my approach was: df.groupby(['name'])[['phase','value']].max()  but it returned incorrect values.

Answer: You don't need to use groupby. Sort values by value and phase (adjust the order if necessary) and drop duplicates by name: out = (df.sort_values(['value', 'phase'], ascending=[False, True]) .drop_duplicates('name') .sort_index(ignore_index=True)) print(out) # Output name phase value 0 BOB 1 0.90 1 JOHN 2 0.45 2 FRANK 3 0.60
Question: I want to generate a 2D numpy array with elements calculated from their positions. Something like the following code: import numpy as np def calculate_element(i, j, other_parameters): # do something return value_at_i_j def main(): arr = np.zeros((M, N)) # (M, N) is the shape of the array for i in range(M): for j in range(N): arr[i][j] = calculate_element(i, j, ...)  This code runs extremely slow since the loops in Python are just not very efficient. Is there any way to do this faster in this case? By the way, for now I use a workaround by calculating two 2D "index matrices". Something like this: def main(): index_matrix_i = np.array([range(M)] * N).T index_matrix_j = np.array([range(N)] * M) ''' index_matrix_i is like [[0,0,0,...], [1,1,1,...], [2,2,2,...], ... ] index_matrix_j is like [[0,1,2,...], [0,1,2,...], [0,1,2,...], ... ] ''' arr = calculate_element(index_matrix_i, index_matrix_j, ...)  Edit1: The code becomes much faster after I apply the "index matrices" trick, so the main question I want to ask is that if there is a way to not use this trick, since it takes more memory. In short, I want to have a solution that is efficient in both time and space. Edit2: Some examples I tested # a simple 2D Gaussian def calculate_element(i, j, i_mid, j_mid, i_sig, j_sig): gaus_i = np.exp(-((i - i_mid)**2) / (2 * i_sig**2)) gaus_j = np.exp(-((j - j_mid)**2) / (2 * j_sig**2)) return gaus_i * gaus_j  # size of M, N M, N = 1200, 4000  # use for loops to go through every element # this code takes ~10 seconds def main_1(): arr = np.zeros((M, N)) # (M, N) is the shape of the array for i in range(M): for j in range(N): arr[i][j] = calculate_element(i, j, 600, 2000, 300, 500) # print(arr) plt.figure(figsize=(8, 5)) plt.imshow(arr, aspect='auto', origin='lower') plt.show()  # use index matrices # this code takes <1 second def main_2(): index_matrix_i = np.array([range(M)] * N).T index_matrix_j = np.array([range(N)] * M) arr = calculate_element(index_matrix_i, index_matrix_j, 600, 2000, 300, 500) # print(arr) plt.figure(figsize=(8, 5)) plt.imshow(arr, aspect='auto', origin='lower') plt.show() 

Answer: You can use np.indices() to generate the desired output: For example, np.indices((3, 4))  outputs: [[[0 0 0 0] [1 1 1 1] [2 2 2 2]] [[0 1 2 3] [0 1 2 3] [0 1 2 3]]]
Question: (ldm) C:\WBC\latent-diffusion-main>python scripts/txt2img.py --prompt "a sunset behind a mountain range, vector image" --ddim_eta 1.0 --n_samples 1 --n_iter 1 --H 384 --W 1024 --scale 5.0 Loading model from models/ldm/text2img-large/model.ckpt Traceback (most recent call last): File "scripts/txt2img.py", line 108, in <module> model = load_model_from_config(config, "models/ldm/text2img-large/model.ckpt") # TODO: check path File "scripts/txt2img.py", line 19, in load_model_from_config model = instantiate_from_config(config.model) File "c:\wbc\latent-diffusion-main\ldm\util.py", line 78, in instantiate_from_config return get_obj_from_str(config["target"])(**config.get("params", dict())) File "c:\wbc\latent-diffusion-main\ldm\util.py", line 86, in get_obj_from_str return getattr(importlib.import_module(module, package=None), cls) File "C:\ProgramData\Anaconda3\envs\ldm\lib\importlib\__init__.py", line 127, in import_module return _bootstrap._gcd_import(name[level:], package, level) File "<frozen importlib._bootstrap>", line 1014, in _gcd_import File "<frozen importlib._bootstrap>", line 991, in _find_and_load File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked File "<frozen importlib._bootstrap>", line 671, in _load_unlocked File "<frozen importlib._bootstrap_external>", line 783, in exec_module File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed File "c:\wbc\latent-diffusion-main\ldm\models\diffusion\ddpm.py", line 12, in <module> import pytorch_lightning as pl File "C:\ProgramData\Anaconda3\envs\ldm\lib\site-packages\pytorch_lightning\__init__.py", line 20, in <module> from pytorch_lightning import metrics # noqa: E402 File "C:\ProgramData\Anaconda3\envs\ldm\lib\site-packages\pytorch_lightning\metrics\__init__.py", line 15, in <module> from pytorch_lightning.metrics.classification import ( # noqa: F401 File "C:\ProgramData\Anaconda3\envs\ldm\lib\site-packages\pytorch_lightning\metrics\classification\__init__.py", line 14, in <module> from pytorch_lightning.metrics.classification.accuracy import Accuracy # noqa: F401 File "C:\ProgramData\Anaconda3\envs\ldm\lib\site-packages\pytorch_lightning\metrics\classification\accuracy.py", line 18, in <module> from pytorch_lightning.metrics.utils import deprecated_metrics, void File "C:\ProgramData\Anaconda3\envs\ldm\lib\site-packages\pytorch_lightning\metrics\utils.py", line 22, in <module> from torchmetrics.utilities.data import get_num_classes as _get_num_classes ImportError: cannot import name 'get_num_classes' from 'torchmetrics.utilities.data' (C:\ProgramData\Anaconda3\envs\ldm\lib\site-packages\torchmetrics\utilities\data.py)  I can't run a latent diffusion neural network. I'm using anaconda and the environment that comes with the neural network. I think I'm doing everything right, but something goes wrong. If you know how to solve this problem, I would be extremely grateful.

Answer: Looking at your error, it appears get_num_classes doesn't exist anymore. I verified this by looking that their github and docs. It was removed after this commit.
Question: I'm trying to visualize my Deep Learning model using visual keras, but i am getting an error which i am not sure i understand. This is my first time using visual keras, and i am not sure what to do. As an example !pip install visual keras import visualkeras import tensorflow as tf tf.keras.utils.plot_model(model, show_shapes=True) input = tf.keras.Input(shape=(100,), dtype='int32', name='input') x = tf.keras.layers.Embedding(output_dim=512, input_dim=10000, input_length=100)(input) x = tf.keras.layers.LSTM(32)(x) x = tf.keras.layers.Dense(64, activation='relu')(x) x = tf.keras.layers.Dense(64, activation='relu')(x) x = tf.keras.layers.Dense(64, activation='relu')(x) output = tf.keras.layers.Dense(1, activation='sigmoid', name='output')(x) model = tf.keras.Model(inputs=[input], outputs=[output]) visualkeras.layered_view(model, legend=True, draw_volume=False)  and the error looks like this TypeError: 'int' object is not iterable. Any help will be much appreciated.

Answer: It is a bug as you can read here. The author suggests to install a newer version of the library: !pip install git+https://github.com/paulgavrikov/visualkeras --upgrade  If for some reason you cannot update the version. Go to the source code of the library (where it was installed) and navigate to visualkeras/layered.py. In line 100 of this file, change z = min(max(z), max_z) to z = min(max([z]), max_z). Save the changes and it will work. An example: !pip install git+https://github.com/paulgavrikov/visualkeras --upgrade import visualkeras import tensorflow as tf input = tf.keras.Input(shape=(100,), dtype='int32', name='input') x = tf.keras.layers.Embedding(output_dim=512, input_dim=10000, input_length=100)(input) x = tf.keras.layers.LSTM(32)(x) x = tf.keras.layers.Dense(64, activation='relu')(x) x = tf.keras.layers.Dense(64, activation='relu')(x) x = tf.keras.layers.Dense(64, activation='relu')(x) output = tf.keras.layers.Dense(1, activation='sigmoid', name='output')(x) model = tf.keras.Model(inputs=[input], outputs=[output]) visualkeras.layered_view(model, legend=True, draw_volume=False)
Question: I instantiate a hydra configuration from a python dataclass. For example from dataclasses import dataclass from typing import Any from hydra.utils import instantiate class Model(): def __init__(self, x=1): self.x = x @dataclass class MyConfig: model: Any param: int static_config = MyConfig(model=Model(x=2), param='whatever') instantiated_config = instantiate(static_config)  Now, I would like to dump this configuration as a yaml, including the _target_ fields that Hydra uses to re-instantiate the objects pointed to inside the configuration. I would like to avoid having to write my own logic to write those _target_ fields, and I imagine there must be some hydra utility that does this, but I can't seem to find it in the documentation.

Answer: See OmegaConf.to_yaml and OmegaConf.save: from omegaconf import OmegaConf # dumps to yaml string yaml_data: str = OmegaConf.to_yaml(my_config) # dumps to file: with open("config.yaml", "w") as f: OmegaConf.save(my_config, f) # OmegaConf.save can also accept a `str` or `pathlib.Path` instance: OmegaConf.save(my_config, "config.yaml")  See also the Hydra-Zen project, which offers automatic generation of OmegaConf objects (which can be saved to yaml).
Question: I want to create a dataset with xarray and want to add attributes to variables while creating the dataset. The xarray documentation provides a way of adding global attribute. For example, as below: ds = xr.Dataset( data_vars=dict( 'temperature'=(["x", "y", "time"], temperature), 'precipitation'=(["x", "y", "time"], precipitation), ), coords=dict( lon=(["x", "y"], lon), lat=(["x", "y"], lat), time=time, reference_time=reference_time, ), attrs=dict(description="Weather related data."),)  One way to add variable attribute would be some like this: ds['temperature'].attrs = {"units": K, '_FillValue': -999}  But, in my opinion it is more like updating the attribute. Is there a way to directly assign attributes while creating the dataset directly using xr.Dataset ?

Answer: Yes, you can directly define variable attributes when defining the data_vars. You just need to provide the attributes in a dictionary Form. See also: https://xarray.pydata.org/en/stable/internals/variable-objects.html In your example above that would be: ds = xr.Dataset( data_vars=dict( temperature=(["x", "y", "time"], temperature,{'units':'K'}), precipitation=(["x", "y", "time"], precipitation,{'units':'mm/day'}), ), coords=dict( lon=(["x", "y"], lon), lat=(["x", "y"], lat), time=time, reference_time=reference_time, ), attrs=dict(description="Weather related data."),)
Question: After I install Google cloud sdk in my computer, I open the terminal and type "gcloud --version" but it says "python was not found" note: I unchecked the box saying "Install python bundle" when I install Google cloud sdk because I already have python 3.10.2 installed. so, how do fix this? Thanks in advance.

Answer: As mentioned in the document:  Cloud SDK requires Python; supported versions are Python 3 (preferred, 3.5 to 3.8) and Python 2 (2.7.9 or later). By default, the Windows version of Cloud SDK comes bundled with Python 3 and Python 2. To use Cloud SDK, your operating system must be able to run a supported version of Python.  As suggested by @John Hanley the CLI cannot find Python which is already installed. Try reinstalling the CLI selecting install Python bundle. If you are still facing the issue another workaround can be to try with Python version 2.x.x . You can follow the below steps : 1.Uninstall all Python version 3 and above. 2.Install Python version -2.x.x (I have installed - 2.7.17) 3.Create environment variable - CLOUDSDK_PYTHON and provide value as C:\Python27\python.exe 4.Run GoogleCloudSDKInstaller.exe again.
Question: I wanted to seek to the start of the file of to write from the start. In the documentation of python 3.9 io.IOBase.seek it is displayed seek has a parameter "whence" yet an error is being displayed:  TypeError: TextIOWrapper.seek() takes no keyword arguments  my code is: with open("t.txt",'a+') as f: f.seek(0,) print(f.readlines()) f.seek(0,whence=0) f.write("12\n23\n32")  I have use "a+" as I want to preserve the contains of the file when it is opened as well edit later. I wanted to edit the contains from the start that's why I used whence = 0, as it would help me edit from start of the stream

Answer: Yeah, it's a little bit weird. Take a look at help(f.seek): Help on built-in function seek: seek(cookie, whence=0, /) method of _io.TextIOWrapper instance  Note the / slash. https://stackoverflow.com/a/24735582/8431111 It says "no keywords, please!". You can specify f.seek(0), or f.seek(0, 0). You just can't name that 2nd parameter whence. It is helpful documentation in the signature, but you can't name it in the call.
Question: I am trying to update my code to the latest Django version, The method is_safe_url() seems to be removed from new version of Django (4.0). I have been looking for an alternative for the past few hours with no luck. Does anyone know of any alternatives for the method in Django 4.0?

Answer: in Django 3.0 they have renamed is_safe_url to url_has_allowed_host_and_scheme. Here you can read more about it docs
Question: When creating 3D scatter plots with matplotlib I noticed that when the alpha (transparency) of the points is varied it will draw them differently depending on how you rotate the view. The example images below are the same plot rotated slightly, which causes the alpha values to mysteriously reverse. Is anyone familiar with this behavior and how to address it? It looks like the 'zorder' (draw order) is a single value for the entire scatter plot call.  Simplified example code to recreate: import matplotlib.pyplot as plt fig = plt.figure() ax = fig.add_subplot(projection="3d") X = [i for i in range(10)] Y = [i for i in range(10)] Z = [i for i in range(10)] S = [(i+1)*400 for i in range(10)] A = [i/10 for i in range(10)] ax.scatter(xs=X, ys=Y, zs=Z, s=S, alpha=A) plt.show()   Python 3.9.5 matplotlib 3.5.1 

Answer: UPDATE: a fix has been added to matplotlib as part of the future 3.11.0 release (not yet released at time of writing) I also posted this on the matplotlib github to see what the developers think. It appears to be a bug with a very low priority, per their latest response: "We have limited core developer resources and mpl_toolkits/mplot3d is only a secondary priority for the core team. Likely this needs someone in the community to pick up and provide a fix." I will try to do so, but am posting this for visibility in the meantime so that other people with more experience are made aware.