eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: So, I'm trying to create an animated plot, but I want the previous frame to be cleared before a new one appears. What I keep getting is all frames at the same time or just a blank plot.  fig, ax = plt.subplots() campo = ax.plot(x2[0], phiSol[0])[0] def init(): campo.set_data([],[]) return campo def update(frame): campo.set_xdata(x2[:frame]) campo.set_ydata(phiSol[:frame]) return campo anima = ani.FuncAnimation(fig=fig, func=update, init_func=init, frames=40, interval=30) HTML(anima.to_jshtml())  I tried to build an init_func, but none of my tries worked. The last try is that one in the above code. How could I do it?

Answer: It's your slicing that's causing the problem. with :frame you're slicing up to frame instead of just grabbing the frame you want. You may have copied an example with a moving point that traces out a line . I just tried this, and it worked how you described.  import numpy as np import matplotlib.pyplot as plt from matplotlib.animation import FuncAnimation # a sin wave, moving through space (test data) x2 = np.linspace(0, 2*np.pi, 100) phiSol = np.array([np.sin(x2 + phase) for phase in np.linspace(0, 2*np.pi, 60)]) fig, ax = plt.subplots() ax.set_xlim(np.min(x2), np.max(x2)) ax.set_ylim(np.min(phiSol), np.max(phiSol)) campo, = ax.plot([], []) def init(): campo.set_data([], []) return campo, def update(frame): campo.set_data(x2, phiSol[frame]) return campo, ani = FuncAnimation(fig, update, frames=len(phiSol), init_func=init, blit=True) # blit=True not necessary, but can optimize the animation. plt.show()  Without the axis limits, mine was showing me an empty window, so you can try removing that and see how it goes, and add something similar if you have problems. I added a couple options to make it a bit better (campo,) to retrieve only the first thing returned by ax.plot instead of a list of objects, and blit=True which can optimize the creation of the animation if it gets slow (see FuncAnimation docs).
Question: I would like this Python 3.10 script (where the pynput code is partially based on this answer) to enter the while loop and at the same time monitor the keys pressed on the keyboard. When q is pressed, I would like it to end. (I do not know threads very well, but the while loop probably should run in the main thread and the keybord monitor should run in a child, concurrent thread). #!/usr/bin/python3 import threading import sys from pynput import keyboard def on_key_press(key): try: k = key.char except: k = key.name if k in ['q']: exit_time = True exit_time = False print("Press q to close.") keyboard_listener = keyboard.Listener(on_press=on_key_press) keyboard_listener.start() keyboard_listener.join() while not exit_time: sleep(1) print("Goodbye") sys.exit(0)  It instead gets locked in an endless wait after keyboard_listener.start(). I don't know if keyboard_listener.join() doesn't run at all, or if it causes the program to lock. However, the while loop is not run. If I end the program with Ctrl+C: ^CTraceback (most recent call last): File "/my/source/./file.py", line 22, in <module> keyboard_listener.join() File "/my/.local/lib/python3.10/site-packages/pynput/_util/__init__.py", line 295, in join super(AbstractListener, self).join(timeout, *args) File "/usr/lib/python3.10/threading.py", line 1096, in join self._wait_for_tstate_lock() File "/usr/lib/python3.10/threading.py", line 1116, in _wait_for_tstate_lock if lock.acquire(block, timeout): KeyboardInterrupt 

Answer: you are joining the listener thread, ie: waiting for it to exit. remove the while loop. the join is already waiting for the thread to exit. also from the docs  Call pynput.keyboard.Listener.stop from anywhere, raise StopException or return False from a callback to stop the listener.  you are waiting for the listener to exit, but you never really tell it to exit, you should probably return False to tell it to exit. def on_key_press(key): try: k = key.char except: k = key.name if k in ['q']: return False # end the listener and unlock the main thread   if you want to do work within the while loop then remove the join, and use threading.Event, as you can wait for it to be signaled instead of sleeping. #!/usr/bin/python3 import threading import sys from pynput import keyboard from time import sleep def on_key_press(key): try: k = key.char except: k = key.name if k in ['q']: thread_exited.set() return False # end the listener and unlock the main thread thread_exited = threading.Event() print("Press q to close.") keyboard_listener = keyboard.Listener(on_press=on_key_press) keyboard_listener.start() # wait for 1 second for the flag to be set, returns true if it was set while not thread_exited.wait(1): print("doing some work ...") keyboard_listener.join() print("Goodbye")
Question: I have the following dataframe: import pandas as pd d = {'Name': ['DataSource', 'DataSource'], 'DomainCode': ['Pr', 'Gov'], 'DomainName': ['Private', 'Government']} df = pd.DataFrame(data=d)  So the dataframe is as follows:  Name DomainCode DomainName 0 DataSource Pr Private 1 DataSource Gov Government  I need to group it by the name to receive two lists:  Name DomainCode DomainName 0 DataSource [Pr, Gov] [Private, Government]  I understand how to do it for a single column: df = df.groupby("Name")["DomainCode"].apply(list).reset_index()  when I receive  Name DomainCode 0 A_DataSource [GOV, PR]  but I cannot add the second column there whatever I tried. How to do this? One more question is that the list returned by the previous command is somehow not a list as it has a length of 1, and not two.

Answer: Please use the following line: df_grouped = df.groupby("Name").agg(list).reset_index()  When you run this line df_grouped = df.groupby("Name")["DomainCode"].apply(list).reset_index()  It returns 1 instead of 2 because Pandas is storing the list as a single string ('[Pr, Gov]') rather than a true Python list. For conversion to real list (from comment): import ast fake_list = "['Pr', 'Gov']" real_list = ast.literal_eval(fake_list) print(real_list) for item in real_list: print(item)  Or: fake_list = "Pr, Gov" real_list = fake_list.split(", ") print(real_list)  Or: import json fake_list = '["Pr", "Gov"]' real_list = json.loads(fake_list) print(real_list)  Or: import re fake_list = "['Pr', 'Gov']" real_list = re.findall(r"'(.*?)'", fake_list) print(real_list)  Output:
Question: When running snakemake on a cluster, and if we don't have specific requirements for some rules about number of cores/memory, then what is the difference between :  Using the classic way, i.e. calling snakemake on the login node, telling it that executor is slurm and that we want X jobs with X cores each, optionally with a profile config file (1 job = 1 rule) Using snakemake like a normal tool, and call it with srun inside a sbatch script without telling it that this is a slurm environment (1 job = whole pipeline)  Example of the second option: #!/bin/bash #SBATCH --job-name=test_sbatch_snakemake #SBATCH --nodes=1 #SBATCH --ntasks=1 #SBATCH --cpus-per-task=64 srun snakemake --cores 64 \ --latency-wait 30 \ --nolock \ --configfile configs/S_with_N.yaml  I have a pipeline where I don't have any specific capacity requirement for my rules (I just want the maximum of them running in parallel), and I think the second option is easier to implement.

Answer: With the second way, your whole workflow will have to wait in the queue before individual rule instances can start. With the first way the resource demand will be spread across the jobs that snakemake will submit, so each rule has a chance to start earlier than what you would have to wait in the second way. (I use a third way: I sbatch a snakemake command that uses slurm as executor. In our cluster, it is considered bad practice to run the main snakemake on the submit/login node. This main snakemake doesn't wait too much in the queue, because it doesn't have to claim a lot of resources.)
Question: Consider the following dataframe example: id date hrz tenor 1 2 3 4 AAA 16/03/2010 2 6m 0.54 0.54 0.78 0.19 AAA 30/03/2010 2 6m 0.05 0.67 0.20 0.03 AAA 13/04/2010 2 6m 0.64 0.32 0.13 0.20 AAA 27/04/2010 2 6m 0.99 0.53 0.38 0.97 AAA 11/05/2010 2 6m 0.46 0.90 0.11 0.14 AAA 25/05/2010 2 6m 0.41 0.06 0.96 0.31 AAA 08/06/2010 2 6m 0.19 0.73 0.58 0.80 AAA 22/06/2010 2 6m 0.40 0.95 0.14 0.56 AAA 06/07/2010 2 6m 0.22 0.74 0.85 0.94 AAA 20/07/2010 2 6m 0.34 0.17 0.03 0.77 AAA 03/08/2010 2 6m 0.13 0.32 0.39 0.95 AAA 16/03/2010 2 1y 0.54 0.54 0.78 0.19 AAA 30/03/2010 2 1y 0.05 0.67 0.20 0.03 AAA 13/04/2010 2 1y 0.64 0.32 0.13 0.20 AAA 27/04/2010 2 1y 0.99 0.53 0.38 0.97 AAA 11/05/2010 2 1y 0.46 0.90 0.11 0.14 AAA 25/05/2010 2 1y 0.41 0.06 0.96 0.31 AAA 08/06/2010 2 1y 0.19 0.73 0.58 0.80 AAA 22/06/2010 2 1y 0.40 0.95 0.14 0.56 AAA 06/07/2010 2 1y 0.22 0.74 0.85 0.94 AAA 20/07/2010 2 1y 0.34 0.17 0.03 0.77 AAA 03/08/2010 2 1y 0.13 0.32 0.39 0.95  How can I grouby the variables id, hrz and tenor and apply the following custom functions across the dates?  def ks_test(x): return scipy.stats.kstest(np.sort(x), 'uniform')[0] def cvm_test(x): n = len(x) i = np.arange(1, n + 1) x = np.sort(x) w2 = (1 / (12 * n)) + np.sum((x - ((2 * i - 1) / (2 * n))) ** 2) return w2  The desired output is the following dataframe (figure results are just examples): id hrz tenor test 1 2 3 4 AAA 2 6m ks_test 0.04 0.06 0.02 0.03 AAA 2 6m cvm_test 0.09 0.17 0.03 0.05 AAA 2 1y ks_test 0.04 0.06 0.02 0.03 AAA 2 1y cvm_test 0.09 0.17 0.03 0.05 

Answer: Use GroupBy.agg with DataFrame.stack for reshape last level of MultiIndex in columns: cols = ['id','hrz', 'tenor'] out = (df.groupby(cols)[df.columns.difference(cols + ['date'], sort=False)] .agg([ks_test, cvm_test]) .rename_axis([None, 'test'], axis=1) .stack(future_stack=True) .reset_index()) print (out) id hrz tenor test 1 2 3 4 0 AAA 2 1y ks_test 0.278182 0.166364 0.254545 0.224545 1 AAA 2 1y cvm_test 0.220803 0.044730 0.158839 0.118321 2 AAA 2 6m ks_test 0.278182 0.166364 0.254545 0.224545 3 AAA 2 6m cvm_test 0.220803 0.044730 0.158839 0.118321  How it working: print (df.groupby(cols)[df.columns.difference(cols +['date'], sort=False)] .agg([ks_test, cvm_test])) 1 2 3 \ ks_test cvm_test ks_test cvm_test ks_test cvm_test id hrz tenor AAA 2 1y 0.278182 0.220803 0.166364 0.04473 0.254545 0.158839 6m 0.278182 0.220803 0.166364 0.04473 0.254545 0.158839 4 ks_test cvm_test id hrz tenor AAA 2 1y 0.224545 0.118321 6m 0.224545 0.118321
Question: I have an array with np.nan values which I want to plot using pcolor. In principle everything works, but I get a warning I cannot get rid of. Using plt.imshow does not give the warning, but I need to specify the x and y coordinates.  MatplotlibDeprecationWarning: Getting the array from a PolyQuadMesh will return the full array in the future (uncompressed). To get this behavior now set the PolyQuadMesh with a 2D array .set_array(data2d).  import numpy as np import matplotlib as mpl import matplotlib.pyplot as plt x = np.linspace(-2,2,100) y = np.linspace(-2,2,100) X, Y = np.meshgrid(x,y) X[X**2+Y**2>4] = np.nan Y[X**2+Y**2>4] = np.nan Z = np.exp(-(X**2+Y**2)) plt.pcolor(Y,X,Z, cmap='viridis') plt.colorbar()  I use matplotlib v3.9.2 and numpy v1.26.4.

Answer: This warning is due to a change in how the internal pcolor logic is structured (changenote here). It is triggered when the colorbar code internally calls get_array on the object returned by pcolor. You can silence the warning by explicitly re-passing your Z array to the set_array method: pc = plt.pcolor(Y,X,Z, cmap='viridis') pc.set_array(Z) plt.colorbar()  Alternatively, you could upgrade your Matplotlib to version 3.10+, since this deprecation is now expired.
Question: I'm trying to follow the solution detailed at this question to prepare a dataset to train a CRNN for HTR (Handwritten Text Recognition). I'm using eScriptorium to adjust text segmentation and transcription, exporting in ALTO format (one XML with text region coordinates for each image) and parsing the ALTO XML to grab the text image regions and export them individually to create a dataset. The problem I'm finding is that I have the region defined at eScriptorium, like this:  But when I apply this code from the selected solution for the above linked question: # Initialize mask mask = np.zeros((img.shape[0], img.shape[1])) # Create mask that defines the polygon of points cv2.fillConvexPoly(mask, pts, 1) mask = mask > 0 # To convert to Boolean # Create output image (untranslated) out = np.zeros_like(img) out[mask] = img[mask]  and display the image I get some parts of the text region filled:  As you can see, some areas that should be inside the mask are filled and, therefore, the image pixels in them are not copied. I've made sure the pixels that make the polygon are correctly parsed and handed to OpenCV to build the mask. I can't find the reason why those areas are filled and I wonder if anyone got into a similar problem and managed to find out the reason or how to avoid it. TIA

Answer: You called cv.fillConvexPoly(). Your polygon is not convex. The algorithm assumed it to be convex and took some shortcuts to simplify the drawing code, so it came out wrong. Use cv.fillPoly() instead. That will draw non-convex polygons correctly. As you point out, the function signatures are not drop-in compatible. fillPoly() works on a list of polygons, while fillComplexPoly() just takes a single polygon. cv.fillConvexPoly(img, points, color) # would be replaced with cv.fillPoly(img, [points], color) # list of one polygon  Each polygon should be a numpy array of shape (N, 1, 2) and it probably needs to be of an integer dtype too, although I'm not sure about that now and it might support floating point dtype in the future.
Question: I am plotting a dataset using a scatter plot in Python, and I am encoding the data both in color and size. I'd like for the legend to represent this. I am aware of .legend_elements(prop='sizes') but I can have either colors or sizes but not both at the same time. I found a way of changing the marker color when using prop='sizes' with th color argument, but that's not really what I intend to do (they are all the same color). Here is a MWE: import pandas as pd import numpy as np import pylab as pl time = pd.DataFrame(np.random.rand(10)) intensity = pd.DataFrame(np.random.randint(1,5,10)) df = pd.concat([time, intensity], axis=1) size = intensity.apply(lambda x: 10*x**2) fig, ax = pl.subplots() scat = ax.scatter(time, intensity, c=intensity, s=size) lgd = ax.legend(*scat.legend_elements(prop="sizes", num=3, \ fmt="{x:.1f}", func=lambda s: np.sqrt(s/10)), \ title="intensity")  and I'd like to have the markers color-coded too. Any help or hint would be appreciated!

Answer: Using legend_elements, you can get the size and a colour-based legend elements separately, then set the colours of the former with the latter. E.g., import pandas as pd import numpy as np import matplotlib.pyplot as pl time = pd.DataFrame(np.random.rand(10)) intensity = pd.DataFrame(np.random.randint(1,5,10)) df = pd.concat([time, intensity], axis=1) size = intensity.apply(lambda x: 10*x**2) fig, ax = pl.subplots() scat = ax.scatter(time, intensity, c=intensity, s=size) # get sized-based legend handles size_handles, text = scat.legend_elements( prop="sizes", num=3, fmt="{x:.1f}", func=lambda s: np.sqrt(s/10) ) # get colour-based legend handles colors = [c.get_color() for c in scat.legend_elements(prop="colors", num=3)[0]] # set colours of the size-based legend handles for i, c in enumerate(colors): size_handles[i].set_color(c) # add the legend lgd = ax.legend(size_handles, text, title="intensity")
Question: Consider the following dataframe example: maturity_date simulation simulated_price realized_price 30/06/2010 1 0.539333333 0.611 30/06/2010 2 0.544 0.611 30/06/2010 3 0.789666667 0.611 30/06/2010 4 0.190333333 0.611 30/06/2010 5 0.413666667 0.611  Apart from setting aside the value of the last column and concatenating, is there any other way to adjust the dataframe such that the last column becomes row? Here is the desired output: maturity_date simulation simulated_price 30/06/2010 1 0.539333333 30/06/2010 2 0.544 30/06/2010 3 0.789666667 30/06/2010 4 0.190333333 30/06/2010 5 0.413666667 30/06/2010 realized_price 0.611 

Answer: Maybe easier is processing dictionary from last row, DataFrame.pop trick is for remove original column realized_price: d = df.iloc[-1].to_dict() d['simulated_price'] = d.pop('realized_price') d['simulation'] = 'realized_price' df.loc[len(df.pop('realized_price'))] = d  Alternative: last = df.columns[-1] d = df.iloc[-1].to_dict() d['simulated_price'] = d.pop(last) d['simulation'] = last df.loc[len(df.pop(last))] = d print (df) maturity_date simulation simulated_price 0 30/06/2010 1 0.539333 1 30/06/2010 2 0.544000 2 30/06/2010 3 0.789667 3 30/06/2010 4 0.190333 4 30/06/2010 5 0.413667 5 30/06/2010 realized_price 0.611000  Another idea is use DataFrame.loc for set new row with default index of DataFrame by select last row in DataFrame.iloc, rename and reappend simulation with new value realized_price in Series.reindex: s = (df.iloc[-1].drop(['simulated_price','simulation']) .rename({'realized_price':'simulated_price'}) .reindex(df.columns[:-1], fill_value='realized_price')) df.loc[len(df.pop('realized_price'))] = s print (df) maturity_date simulation simulated_price 0 30/06/2010 1 0.539333 1 30/06/2010 2 0.544000 2 30/06/2010 3 0.789667 3 30/06/2010 4 0.190333 4 30/06/2010 5 0.413667 5 30/06/2010 realized_price 0.611000  Alternative is first reassign column simulation, then get last row and processing Series: s = (df.assign(simulation='realized_price') .iloc[-1] .drop(['simulated_price']) .rename({'realized_price':'simulated_price'})) df.loc[len(df.pop('realized_price'))] = s print (df) maturity_date simulation simulated_price 0 30/06/2010 1 0.539333 1 30/06/2010 2 0.544000 2 30/06/2010 3 0.789667 3 30/06/2010 4 0.190333 4 30/06/2010 5 0.413667 5 30/06/2010 realized_price 0.611000  Another idea with concat: out = (pd.concat([df, df.iloc[[-1]] .assign(simulation='realized_price', simulated_price=df['realized_price'].iat[0])], ignore_index=True) .drop('realized_price', axis=1)) print (out) maturity_date simulation simulated_price 0 30/06/2010 1 0.539333 1 30/06/2010 2 0.544000 2 30/06/2010 3 0.789667 3 30/06/2010 4 0.190333 4 30/06/2010 5 0.413667 5 30/06/2010 realized_price 0.611000
Question: I'm using Django with django-autocomplete-light and Select2 to create an autocomplete field. The Select2 field is dynamically added to the page when another field is selected. It fetches data from a Django autocomplete view, and everything works fine. Now, I need to filter the queryset in my autocomplete view based on an extra parameter (argId). However, I'm not sure how to pass this parameter correctly. JavaScript (Select2 Initialization) function getElement(argId) { let elementSelect = $("<select></select>"); let elementDiv = $(`<div id='element_id' style='text-align: center'></div>`); elementDiv.append(elementSelect); $(elementSelect).select2({ ajax: { url: "/myautocomplete/class", data: function (params) { return { q: params.term, // Search term arg_id: argId // Pass extra parameter }; }, processResults: function (data) { return { results: data.results // Ensure correct format }; } }, placeholder: "Element...", minimumInputLength: 3 }); return elementDiv;  } Django Autocomplete View class ElementAutocomplete(LoginRequiredMixin, autocomplete.Select2QuerySetView): def get_queryset(self): qs = MyModel.objects.filter(...)  I want to pass argId from JavaScript to the Django view so that the queryset is filtered accordingly. However, I am not sure if my approach is correct or how to achieve this. Appreciate any suggestions or improvements. Thanks!

Answer: Just pass them as query params /myautocomplete/class?title=title_1 and you can catch them in the class class ElementAutocomplete(LoginRequiredMixin, autocomplete.Select2QuerySetView): def get_queryset(self): title = self.request.GET.get("title") qs = MyModel.objects.all() if title is not None: qs.filter(title__icontains=title)
Question: Description:  Input is a CSV file  CSV file contains columns of different data types: Ordinal Values, Nominal Values, Numerical Values and Multi Value  For the multivalue columns. Minimum is 1, maximum is 5 values. The input is similar to this:      Job Perks Insurance Benefits     Online Courses; Certification Programs; Cross Training Life Insurance; Dental Insurance   Leadership Development Programs; Online Courses Life Insurance; Accident Insurance     Multivalue Expected Output:     Job Perks_Online Courses Job Perks_Certification Programs Job Perks_Cross Training Job Perks_Leadership Development Programs Insurance Benefits_Life Insurance Insurance Benefits_Dental Insurance Insurance Benefits_Accident Insurance     1 1 1 0 1 1 0   1 0 0 1 1 0 1     How do I preprocess the CSV input and save it to a dataframe with the above expected output? I am able to preprocess nominal attributes to the expected output(sample code below), but finding it hard to convert multivalues  Input: CSV Dataset: https://github.com/omnislayer/WorkDataSet/blob/main/ECP_Unedited.csv Sample Code: #For the nominal: import pandas as pd import numpy as np import dtale #Better STDOUT for dataframes nominalColumns = ["Gender", "Marital Status", "Educational Attainment", "Employment Status", "Company Bonus Structure", "Company Medical Plan Type"] multivalueColumns = ["Job Perks", "Professional Development Opportunities", "Insurance Benefits"] df = pd.read_csv('ECP_Unedited.csv') #Convert Nominal Columns newCols = pd.get_dummies(df[nominalColumns], dtype=int) df = df.drop(columns=nominalColumns) df = pd.concat([df, newCols], axis=1) dtale.show(df) #Convert Multivalue Columns #INSERT CODE HERE! 

Answer: You could combine str.get_dummies, add_prefix, and pd.concat with a generator: out = pd.concat( ( df[col].str.get_dummies(sep='; ').add_prefix(f'{col}_') for col in multivalueColumns ), axis=1, )  Output:  Job Perks_Certification Programs Job Perks_Cross Training Job Perks_Leadership Development Programs Job Perks_Online Courses Insurance Benefits_Accident Insurance Insurance Benefits_Dental Insurance Insurance Benefits_Life Insurance 0 1 1 0 1 0 1 1 1 0 0 1 1 1 0 1