eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: macOS 12.3 update drops Python 2 and replaces it with version 3: https://developer.apple.com/documentation/macos-release-notes/macos-12_3-release-notes  Python Deprecations Python 2.7 was removed from macOS in this update. Developers should use Python 3 or an alternative language instead. (39795874)  I understand we need to migrate to version 3, but in the meantime we still need version 2. Homebrew does not seem to have it anymore: brew install python@2.7 Warning: No available formula with the name "python@2.7". Did you mean python@3.7, python@3.9, python@3.8, python@3.10 or python-yq? brew install python2 Warning: No available formula with the name "python2". Did you mean ipython, bpython, jython or cython?  What gives?

Answer: You can get any Python release, including the last Python 2, from the official download site: https://www.python.org/downloads/release/python-2718/ → macOS 64-bit installer
Question: Here I have some lists with any particular element (a, b, and c) a = [2, 4, [9, 10]] b = [1, 3, 5, 9, [11, 13, 14, 15, [16, 17, 19, 24]]] c = [2, 4, [5, 11, 13, [14, 17, 29, [31, 19]]]] npA = np.array(a, dtype = object) npB = np.array(b, dtype = object) npC = np.array(c, dtype = object)  I am trying to get the deepest list in each list a, b, c print(npA[-1]) # it shows [9,10] print(npB[-1][-1]) # it shows [16, 17, 19, 24] print(npC[-1][-1][-1]) # it shows [31, 19]  How do get the generalization of this problem? or is there any built-in function of NumPy that can directly handle this?

Answer: You can solve this recursively, without numpy: from typing import List def deepest_list(l: List) -> List: last_item = l[-1] if isinstance(last_item, list): return deepest_list(last_item) return l  output: deepest_list(c) [31, 19]
Question: I have the following code with multiple cases: def _extract_property_value(selector: Selector) -> str: raw_value = selector.xpath("span[2]") default_value = raw_value.xpath("./text()").get().strip() value_with_a = ', '.join([value.strip() for value in raw_value.xpath("./a /text()").getall()]) value_with_div_and_a = ', '.join([value.strip() for value in raw_value.xpath("./div /a /text()").getall()]) if value_with_a: return value_with_a elif value_with_div_and_a: return value_with_div_and_a elif default_value: return default_value  I want to get rid of if-statements and simplify this code as much as it is possible. I am not good Pyton dev. I know there is pattern "strategy" and below I was trying to implement that: def _extract_property_value(selector: Selector) -> str: raw_value = selector.xpath("span[2]") values_dict = { 'default value': raw_value.xpath("./text()").get().strip(), 'value with a': ', '.join([value.strip() for value in raw_value.xpath("./a /text()").getall()]), 'value with div and a': ', '.join([value.strip() for value in raw_value.xpath("./div /a /text()").getall()]) } return [item for item in values_dict.values() if item != ''][0]  but... Now when I think of this - that was bad idea to use strategy there. I am not sure. Can someone help me to simplify that code? Or those if-statements are just necessary.

Answer: We can reduce the number of if statements but without the aid of a dictionary. The code in question unconditionally assigns values to 3 variables. Having done so, those variables are examined to determine which, if any, is to be returned to the caller. However, there are no dependencies between those variables. Therefore they could be processed in order of priority and, if an appropriate value is acquired, then it could be returned immediately thereby making the process considerably more efficient. def _extract_property_value(selector): def f1(rv): return rv.xpath("./text()").get().strip() def f2(rv): return ', '.join([value.strip() for value in rv.xpath("./a /text()").getall()]) def f3(rv): return ', '.join([value.strip() for value in rv.xpath("./div /a /text()").getall()]) raw_value = selector.xpath("span[2]") for func in [f2, f3, f1]: # note the priority order - default last if (v := func(raw_value)): return v  The revised function will implicitly return None if suitable values are not found. In this respect it is no different to the OP's original code
Question: I have a list of excel files (.xlsx,.xls), I'm trying to get headers of each of these files after loaded. Here I have taken a one excel file and loaded into pandas as. pd.read_excel("sample.xlsx")  output is:  Here we would like to get an header information as per our requirement, here in the attached image the required headers are existed at index 8 as you can see in red color coded. pd.read_excel('sample.xlsx',skiprows=9)  as we know now we have a correct header at 8 i can go back and specify in read_excel as skip_rows at 8 so that it reads from this index and headers will be appeared as.  How to handle this type of cases programmatically among a list of excel files where we don't know where the header is existed? in this case we have known that header is at 8. but what if we don't know this in other files. Sample file can be downloaded for your ref: https://github.com/myamullaciencia/pg_diploma_ai_ml_uohyd/blob/main/sample_file.xlsx

Answer: Use: df = pd.read_excel('sample_file.xlsx') #test all rows if previous row is only NaNs m1 = df.shift(fill_value=0).isna().all(axis=1) #test all rows if no NaNs m2 = df.notna().all(axis=1) #chain together and filter all next rows after first match df = df[(m1 & m2).cummax()] #set first row to columns names df = df.set_axis(df.iloc[0].rename(None), axis=1).iloc[1:].reset_index(drop=True) print (df) LN FN SSN DOB DOH Gender Comp_2011 Comp_2010 \ 0 Ax Bx 000-00-0000 8/3/1800 1/1/1800 Male 384025.56 396317 1 Er Ds 000-00-0000 5/7/1800 7/1/1800 Male 382263.86 392474 2 Po Ch 000-00-0000 9/9/1800 1/1/1800 Male 406799.34 395677 3 Rt Da 000-00-0000 6/24/1800 7/1/1800 Male 395767.12 424093 4 Yh St 000-00-0000 3/15/1800 7/1/1800 Male 376936.58 373754 5 Ws Ra 000-00-0000 6/12/1800 7/10/1800 Male 425720.06 420927 Comp_2009 Allocation Group NRD 0 360000 0.05 2022-09-01 00:00:00 1 360000 0.05 2015-06-01 00:00:00 2 360000 0.05 2013-01-01 00:00:00 3 360000 0.05 2020-07-01 00:00:00 4 360000 0 2013-01-01 00:00:00 5 306960 0 2034-07-01 00:00:00
Question: I have a 20 x 20 square matrix. I want to take the first 2 rows and columns out of every 5 rows and columns, which means the output should be a 8 x 8 square matrix. This can be done in 2 consecutive steps as follows: import numpy as np m = 5 n = 2 A = np.arange(400).reshape(20,-1) B = np.asarray([row for i, row in enumerate(A) if i % m < n]) C = np.asarray([col for j, col in enumerate(B.T) if j % m < n]).T  However, I am looking for efficiency. Is there a more Numpyic way to do this? I would prefer to do this in one step.

Answer: You can use np.ix_ to retain the elements whose row / column indices are less than 2 modulo 5: import numpy as np m = 5 n = 2 A = np.arange(400).reshape(20,-1) mask = np.arange(20) % 5 < 2 result = A[np.ix_(mask, mask)] print(result)  This outputs: [[ 0 1 5 6 10 11 15 16] [ 20 21 25 26 30 31 35 36] [100 101 105 106 110 111 115 116] [120 121 125 126 130 131 135 136] [200 201 205 206 210 211 215 216] [220 221 225 226 230 231 235 236] [300 301 305 306 310 311 315 316] [320 321 325 326 330 331 335 336]]
Question: What are these zombie ipykernel_launcher process in my machine, which are hogging to much memory:  This is output of htop command, but I ps for those processes,(to kill them) I do not see them as:  ps -ef|grep ipykernel  Not sure, how to get rid of these memory hogs!

Answer: The reason why you're seeing all these processes in htop and not with ps is that htop is showing threads (see https://serverfault.com/questions/24198/why-does-htop-show-lots-of-apache2-processes-by-ps-aux-doesnt). Type "-H" inside htop to toggle showing threads. Automatically stop idle kernels Concerning Jupyter notebook processes in general: kernels are small computational engines and consume a lot of resources (mainly memory) even when they're not active. This is why one should encourage users to stop running kernels when they're not in use. The problem is that even if one closes a tab or the whole browser, the kernel keeps running, so one forgets about the kernels! Since it's unlikely that users will shutdown their kernels, consider stopping idle kernels by configuring the parameter NotebookApp.shutdown_no_activity_timeoutInt in your Jupyter configuration file jupyter_notebook_config.py.  NotebookApp.shutdown_no_activity_timeoutInt. Default: 0 Shut down the server after N seconds with no kernels or terminals running and no activity. This can be used together with culling idle kernels (MappingKernelManager.cull_idle_timeout) to shutdown the notebook server when it’s not in use. This is not precisely timed: it may shut down up to a minute later. 0 (the default) disables this automatic shutdown.  See also these properties: # shutdown the server after no activity for an hour c.ServerApp.shutdown_no_activity_timeout = 60 * 60 # shutdown kernels after no activity for 20 minutes c.MappingKernelManager.cull_idle_timeout = 20 * 60 # check for idle kernels every two minutes c.MappingKernelManager.cull_interval = 2 * 60  If that doesn't work, you may need to run a cron job to kill the ipykernel processes with kill after a certain amount of elapsed time (see for instance https://unix.stackexchange.com/questions/531040/list-processes-that-have-been-running-more-than-2-hours). One-time solution A quick solution to solve the problem is to restart the Jupyter notebook/Jupyter Hub. This will stop all kernels.
Question: I have a list of four points of a rotated rectangle in the form of: points = [[x1, y1], [x2, y2], [x3, y3], [x4, y4]]  I can crop in PIL using: img.crop((x1, y1, x2, y2))  But this doesn't work with a rotated rectangle. Just to clarify I want the resulting cropped image to be rotated so the cropped area becomes a non-rotated rectangle. I am willing to use openCV, although would like to avoid it as the conversion of an image from PIL to openCV takes time, and I will be itterating through this process about 100 times.

Answer: If you start with this image:  You can do it like this using a QuadTransform: #!/usr/bin/env python3 from PIL import Image, ImageTransform # Open starting image and ensure RGB im = Image.open('a.png').convert('RGB') # Define 8-tuple with x,y coordinates of top-left, bottom-left, bottom-right and top-right corners and apply transform=[31,146,88,226,252,112,195,31] result = im.transform((200,100), ImageTransform.QuadTransform(transform)) # Save the result result.save('result.png')
Question: I want to replace the missing values from a column with people's ages (which also contains numerical values, not only NaN values) but everything I've tried so far either doesn't work how I want it to or it doesn't work at all. I wish to apply a random variable generator which follows a normal distribution using the mean and standard deviation obtained with that column. I have tried the following:  Replacing with numpy, replaces NaN values but with the same number for all of them df_travel['Age'] = df_travel['Age'].replace(np.nan, round(rd.normalvariate(age_mean, age_std),0))   Fillna with pandas, also replaces NaN values but with the same number for all of them df_travel['Age'] = df_travel['Age'].fillna(round(rd.normalvariate(age_mean, age_std),0))   Applying a function on the dataframe with pandas, replaces NaN values but also changes all existing numerical values (I only wish to fill the NaN values) df_travel['Age'] = df_travel['Age'].where(df_travel['Age'].isnull() == True).apply(lambda v: round(rd.normalvariate(age_mean, age_std),0))    Any ideas would be appreciated. Thanks in advance.

Answer: Series.fillna can accept a Series, so generate a random array of size len(df_travel): rng = np.random.default_rng(0) mu = df_travel['Age'].mean() sd = df_travel['Age'].std() filler = pd.Series(rng.normal(loc=mu, scale=sd, size=len(df_travel))) df_travel['Age'] = df_travel['Age'].fillna(filler)
Question: Suppose I had a function call as a string, like "log(2, floor(9.4))". I want to deconstruct the call in a way that allows me to access the function name and arguments for the firstmost call and accurately deducts whether a function call as an argument is an argument or not. For example, the arguments when deconstructing the string above would come to [2, floor(9.4)] I've already tried to use some string parsing techniques (e.g. splitting on commas), but it doesn't appear to be working.

Answer: You can use the ast module: import ast data = "log(2, floor(9.4))" parse_tree = ast.parse(data) # ast.unparse() is for 3.9+ only. # If using an earlier version, use the astunparse package instead. result = [ast.unparse(node) for node in parse_tree.body[0].value.args] print(result)  This outputs: ['2', 'floor(9.4)']  I pulled the value to iterate over from manually inspecting the output of ast.dump(parse_tree). Note that I've written something a bit quick and dirty, since there's only one string to parse. If you're looking to parse a lot of these strings (or a larger program), you should create a subclass of ast.NodeVisitor. If you want to also make modifications to the source code, you should create a subclass of ast.NodeTransformer instead.
Question: Is it possible to make CPU only reductions with JAX comparable to Numba in terms of computation time? The compilers come straight from conda: $ conda install -c conda-forge numba jax  Here is a 1-d NumPy array example import numpy as np import numba as nb import jax as jx @nb.njit def reduce_1d_njit_serial(x): s = 0 for xi in x: s += xi return s @jx.jit def reduce_1d_jax_serial(x): s = 0 for xi in x: s += xi return s N = 2**10 a = np.random.randn(N)  Using timeit on the following  np.add.reduce(a) gives 1.99 µs ... reduce_1d_njit_serial(a) gives 1.43 µs ... reduce_1d_jax_serial(a).item() gives 23.5 µs ...  Note that jx.numpy.sum(a) and using jx.lax.fori_loop gives comparable (marginally slower) comp. times to reduce_1d_jax_serial. It seems there is a better way to craft the reduction for XLA. EDIT: compile times were not included as a print statement proceeded to check results.

Answer: When performing these kinds of microbenchmarks with JAX, you have to be careful to ensure you're measuring what you think you're measuring. There are some tips in the JAX Benchmarking FAQ. Implementing some of these best practices, I find the following for your benchmarks: import jax.numpy as jnp # Native jit-compiled XLA sum jit_sum = jx.jit(jnp.sum) # Avoid including device transfer cost in the benchmarks a_jax = jnp.array(a) # Prevent measuring compilation time _ = reduce_1d_njit_serial(a) _ = reduce_1d_jax_serial(a_jax) _ = jit_sum(a_jax) %timeit np.add.reduce(a) # 100000 loops, best of 5: 2.33 µs per loop %timeit reduce_1d_njit_serial(a) # 1000000 loops, best of 5: 1.43 µs per loop %timeit reduce_1d_jax_serial(a_jax).block_until_ready() # 100000 loops, best of 5: 6.24 µs per loop %timeit jit_sum(a_jax).block_until_ready() # 100000 loops, best of 5: 4.37 µs per loop  You'll see that for these microbenchmarks, JAX is a few milliseconds slower than both numpy and numba. So does this mean JAX is slow? Yes and no; you'll find a more complete answer to that question in JAX FAQ: is JAX faster than numpy?. The short summary is that this computation is so small that the differences are dominated by Python dispatch time rather than time spent operating on the array. The JAX project has not put much effort into optimizing for Python dispatch of microbenchmarks: it's not all that important in practice because the cost is incurred once per program in JAX, as opposed to once per operation in numpy.
Question: I'm new to python and currently is working on a project that requires me to export pandas data frame from google collab to a google spreadsheet with multiple tabs. Previously when I run this specific code, there are no errors but then now it shows an error like this:  TypeError Traceback (most recent call last) <ipython-input-74-c8b829c43616> in <module>() 5 gauth.credentials = GoogleCredentials.get_application_default() 6 drive = GoogleDrive(gauth) ----> 7 gc = gspread.authorize(GoogleCredentials.get_application_default()) 2 frames /usr/local/lib/python3.7/dist-packages/gspread/utils.py in convert_credentials(credentials) 59 60 raise TypeError( ---> 61 'Credentials need to be from either oauth2client or from google-auth.' 62 ) 63 TypeError: Credentials need to be from either oauth2client or from google-auth.  Here is the code that I use to create authentication. #Import PyDrive and associated libraries. #This only needs to be done once per notebook. from pydrive.auth import GoogleAuth from pydrive.drive import GoogleDrive from google.colab import auth from oauth2client.client import GoogleCredentials import gspread #Authenticate and create the PyDrive client. #This only needs to be done once per notebook. auth.authenticate_user() gauth = GoogleAuth() gauth.credentials = GoogleCredentials.get_application_default() drive = GoogleDrive(gauth) gc = gspread.authorize(GoogleCredentials.get_application_default())  Any help would be much appreciated.

Answer: I had the same problem today and found this answer: https://github.com/burnash/gspread/issues/1014#issuecomment-1082536016 I finally solved it by replacing the old code with this one: from google.colab import auth auth.authenticate_user() import gspread from google.auth import default creds, _ = default() gc = gspread.authorize(creds)
Question: I'm trying to make a currency recognition model and I did so using a dataset on kaggle and colab using yolov5 and I exactly carried out the steps explained on yolov5 github. At the end, I downloaded a .pt file which has the weights of the model and now I want to use it in python file to detect and recognize currency . How to do this? I am a beginner in computer vision and I am totally confused about what to do. I am searching over and over but I don't reach anything. import torch # Model model=torch.load('E:\_best.pt') # Images imgs=['E:\Study\currency.jpg'] # Inference results = model(imgs) # Results results.print() results.save() # or .show() results.show() results.xyxy[0] # img1 predictions (tensor) results.pandas().xyxy[0] 

Answer: If you want to read trained parameters from .pt file and load it into your model, you could do the following. file = "model.pt" model = your_model() model.load_state_dict(torch.load(file)) # this will automatically load the file and load the parameters into the model.  before calling load_state_dict(), be sure that the .pt file contains only model parameters, otherwise, error occurs. This can be checked by print(torch.load(file)).
Question: I want to convert all boolean columns in my pandas dataframe into 0 and 1 by using pd.get_dummies. However, the boolean values stay the same after the get_dummies function. For example: tmp = pd.DataFrame([ ['green' , True], ['red' , False], ['blue' , True]]) tmp.columns = ['color', 'class'] pd.get_dummies(tmp) # I have also tried pd.get_dummies(tmp, dtype=int), but got the same output  I got:  class color_blue color_green color_red 0 True 0 1 0 1 False 0 0 1 2 True 1 0 0  but I need:  color_blue color_green color_red class_True class_False 0 0 1 0 1 0 1 0 0 1 0 1 2 1 0 0 1 0  update: My dataframe includes numeric data so convert all columns in the dataframe into string may not be the best soluion.

Answer: For processing boolean columns convert them to strings (here are converted all columns): print (pd.get_dummies(tmp.astype(str))) color_blue color_green color_red class_False class_True 0 0 1 0 0 1 1 0 0 1 1 0 2 1 0 0 0 1  Of convert only boolean: print (pd.get_dummies(tmp.astype({'class':'str'}))) color_blue color_green color_red class_False class_True 0 0 1 0 0 1 1 0 0 1 1 0 2 1 0 0 0 1  You can create dictionary only for boolean columns: d = dict.fromkeys(tmp.select_dtypes('bool').columns, 'str') print (pd.get_dummies(tmp.astype(d)))
Question: Today (2022/3/31) I let the auto-update function update my VS Code to latest version 1.66.0 on Windows. After that, my normal debugging process doesn't work any more: when I press F5, the debugging control panel flashes and disappears immediately, nothing else happends. I couldn't find any useful error message on output and terminal windows. My launch.json file looks like this: { "name": "DEBUG", "type": "python", "request": "launch", "program": "${workspaceFolder}\\starting.py", "console": "integratedTerminal", "justMyCode": false, }  I tried to change the console above to externalTerminal but it didn't help. Could someone tell me how to find out what's going on here? Cheers,

Answer: You can try to reinstall the Python extension or install the older version of the Python extension. Delete the deprecated configuration of python.pythonPath in your settings.json.
Question: Compiling python2 in vscode gives an error. But when I compile python3 it succeeds. print('test')  returns: ImportError: No module named _thread PS C:\source> c:; cd 'c:\source'; & 'C:\Python27\python.exe' 'c:\Users\keinblue\.vscode\extensions\ms-python.python-2022.4.0\pythonFiles\lib\python\debugpy\launcher' '52037' '--' 'c:\source\test.py' Traceback (most recent call last): File "C:\Python27\lib\runpy.py", line 174, in _run_module_as_main "__main__", fname, loader, pkg_name) File "C:\Python27\lib\runpy.py", line 72, in _run_code exec code in run_globals File "c:\Users\keinblue\.vscode\extensions\ms-python.python-2022.4.0\pythonFiles\lib\python\debugpy\__main__.py", line 43, in <module> from debugpy.server import cli File "c:\Users\keinblue\.vscode\extensions\ms-python.python-2022.4.0\pythonFiles\lib\python\debugpy/../debugpy\server\__init__.py", line 9, in <module> import debugpy._vendored.force_pydevd # noqa File "c:\Users\keinblue\.vscode\extensions\ms-python.python-2022.4.0\pythonFiles\lib\python\debugpy/../debugpy\_vendored\force_pydevd.py", line 37, in <module> pydevd_constants = import_module('_pydevd_bundle.pydevd_constants') File "C:\Python27\lib\importlib\__init__.py", line 37, in import_module __import__(name) File "c:\Users\keinblue\.vscode\extensions\ms-python.python-2022.4.0\pythonFiles\lib\python\debugpy\_vendored\pydevd\_pydevd_bundle\pydevd_constants.py", line 362, in <module> from _pydev_bundle._pydev_saved_modules import thread, threading File "c:\Users\keinblue\.vscode\extensions\ms-python.python-2022.4.0\pythonFiles\lib\python\debugpy\_vendored\pydevd\_pydev_bundle\_pydev_saved_modules.py", line 94, in <module> import _thread as thread; verify_shadowed.check(thread, ['start_new_thread', 'start_new', 'allocate_lock']) ImportError: No module named _thread 

Answer: There is an issue with the vscode python extension version 2022.4.0 just downgrade to version 2022.2.1924087327 and it will work as it works for me now Just follow these steps:  Go to extensions. Click on Gear Icon for the installed extension Click on Install Another Version select the version you wish to install