eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: What would be the correct type hint for a function that accepts an one dimensional array-like object? More specifically, my function uses np.percentile and I would like to 'match' np.percentile's flexibility in terms of the kind of array it accepts (List, pandas Series, numpy array, etc.). Below illustrates what I'm looking for: def foo(arr: array-like) -> float: p = np.percentile(arr, 50) return p 

Answer: Use numpy.typing.ArrayLike: from numpy.typing import ArrayLike def foo(arr: ArrayLike) -> float: p = np.percentile(arr, 50) return p
Question: I have a list as follows, flat_list = ['hello,5', 'mellow,4', 'mellow,2', 'yellow,2', 'yellow,7', 'hello,7', 'mellow,7', 'hello,7']  I would like to get the sum of the values if they share the same word, so the output should be, desired output: l = [('hello',19), ('yellow', 9), ('mellow',13)]  so far, I have tried the following, new_list = [v.split(',') for v in flat_list] d = {} for key, value in new_list: if key not in d.keys(): d[key] = [key] d[key].append(value) # getting rid of the first key in value lists val = [val.pop(0) for k,val in d.items()] # summing up the values va = [sum([int(x) for x in va]) for ka,va in d.items()]  however for some reason the last sum up does not work and i do not get my desired output

Answer: Here is a variant for accomplishing your goal using defaultdict: from collections import defaultdict t = ['hello,5', 'mellow,4', 'mellow,2', 'yellow,2', 'yellow,7', 'hello,7', 'mellow,7', 'hello,7'] count = defaultdict(int) for name_number in t: name, number = name_number.split(",") count[name] += int(number)  You could also use Counter: from collections import Counter count = Counter() for name_number in t: name, number = name_number.split(",") count[name] += int(number)  In both cases you can convert the output to a list of tuples using: list(count.items()) # -> [('hello', 19), ('mellow', 13), ('yellow', 9)]   I ran your code and I do get the correct results (although not in your desired format).
Question: I would like to color in red cells of a DataFrame on one column, based on the value of another column. Here is an example: df = pd.DataFrame([ { 'color_A_in_red': True , 'A': 1 }, { 'color_A_in_red': False , 'A': 2 }, { 'color_A_in_red': True , 'A': 2 }, ])  should give:  I know how to color a cell of a df in red but only based on the value of this cell, not the value of another cell: df_style = df.style df_style.applymap(func=lambda x: 'background-color: red' if x == 2 else None, subset=['A']) df_style   Is there a way to color cells of a DataFrame based on the value of another column ?

Answer: Use custom function for DataFrame of styles is most flexible solution here: def highlight(x): c = f"background-color:red" #condition m = x["color_A_in_red"] # DataFrame of styles df1 = pd.DataFrame('', index=x.index, columns=x.columns) # set columns by condition df1.loc[m, 'A'] = c return df1 df.style.apply(highlight, axis=None)
Question: What would be the correct way to format the name of a type alias—intended to be local to its module—according to the PEP8 style guide? # mymodule.py from typing import TypeAlias mytype: TypeAlias = int def f() -> mytype: return mytype() def g() -> mytype: return mytype()  Should mytype be formatted in CapWords because it introduces a new type similar to creating new classes? Or, should mytype be formatted in all caps because it is treated similarly to a constant? Is there a way to differentiate between type aliases that will remain unchanged (constant) throughout the lifetime of the program and ones that can change (similar to the Final annotation for constants)? Also, should mytype be prefixed with an underscore (as in _mytype) to indicate that the type alias shouldn't be used outside this module?

Answer: The PEP Style Guide does not have any explicit guidance on how to format TypeAliases. The guide does contain some rules on type variables, but that's not quite what you're asking for.  The next best resource I could find was Google's Python Style Guide, which does happen to contain some guidance on how to name TypeAliases:  3.19.6 Type Aliases You can declare aliases of complex types. The name of an alias should be CapWorded. If the alias is used only in this module, it should be _Private. For example, if the name of the module together with the name of the type is too long: _ShortName = module_with_long_name.TypeWithLongName ComplexMap = Mapping[str, List[Tuple[int, int]]]  Other examples are complex nested types and multiple return variables from a function (as a tuple).  Under this, the name of your type alias should be MyType if used across multiple modules, or _MyType if only used in the module that it is declared in.  With all of this being said, remember that consistency with the existing codebase is what's most important. As the PEP style guide states:  A style guide is about consistency. Consistency with this style guide is important. Consistency within a project is more important. Consistency within one module or function is the most important.
Question: In the command line I am used to run/create containers with specific GPUs using the --gpus argument: docker run -it --gpus '"device=0,2"' ubuntu nvidia-smi The Docker SDK for Python documentation was not very helpful and I could not find a good explanation on how to do the same with the python SDK. Is there a way to do it?

Answer: This is how you can run/create docker containers with specific GPUs using the Docker SDK for Python: client.containers.run('ubuntu', "nvidia-smi", device_requests=[ docker.types.DeviceRequest(device_ids=['0','2'], capabilities=[['gpu']])])  This way you can also use other GPU resource options specified here: https://docs.docker.com/config/containers/resource_constraints/
Question: I tried to install many dependencies for a virtual environment using poetry. When it gets to gevent (20.9.0) it gets the following import error: ImportError: dlopen(/private/var/folders/21/wxg5bdsj1w3f3j_9sl_pktbw0000gn/T/pip-build-env-50mwte36/overlay/lib/python3.8/site-packages/_cffi_backend.cpython-38-darwin.so, 0x0002): tried: '/private/var/folders/21/wxg5bdsj1w3f3j_9sl_pktbw0000gn/T/pip-build-env-50mwte36/overlay/lib/python3.8/site-packages/_cffi_backend.cpython-38-darwin.so' (mach-o file, but is an incompatible architecture (have 'x86_64', need 'arm64e')), '/usr/local/lib/_cffi_backend.cpython-38-darwin.so' (no such file), '/usr/lib/_cffi_backend.cpython-38-darwin.so' (no such file)  I've tried to use pip3 instead, but still had the same problem.

Answer: I've have this problem with other libraries also and this solution worked some times: arch -arm64 <poetry or pip> install <lib to istall>  Using arch -arm64 allowed me to install the rigt wheel for the M1 processor
Question: How to add new feature like length of data frame & Drop rows value using indexing. I want to a add a new column where I can count the no-of rows available in a data frame, & using indexing drop rows value. for i in range(len(df)): if (df['col1'][i] == df['col2'][i]) and (df['col4'][i] == df['col3'][i]): pass elif (df['col1'][i] == df['col3'][i]) and (df['col4'][i] == df['col2'][i]): df['col1'][i] = df['col2'][i] df['col4'][i] = df['col3'][i] else: df = df.drop(i) 

Answer: Polars doesn't allow much mutation and favors pure data handling. Meaning that you create a new DataFrame instead of modifying an existing one. So it helps to think of the data you want to keep instead of the row you want to remove. Below I have written an example that keeps all data except for the 2nd row. Note that the slice will be the fastest of the two and will have zero data copy. df = pl.DataFrame({ "a": [1, 2, 3], "b": [True, False, None] }).with_row_index() print(df) # filter on condition df_a = df.filter(pl.col("index") != 1) # stack two slices df_b = df[:1].vstack(df[2:]) # or via explicit slice syntax # df_b = df.slice(0, 1).vstack(df.slice(2, -1)) assert df_a.equals(df_b) print(df_a)  Outputs: shape: (3, 3) ┌───────┬─────┬───────┐ │ index ┆ a ┆ b │ │ --- ┆ --- ┆ --- │ │ u32 ┆ i64 ┆ bool │ ╞═══════╪═════╪═══════╡ │ 0 ┆ 1 ┆ true │ │ 1 ┆ 2 ┆ false │ │ 2 ┆ 3 ┆ null │ └───────┴─────┴───────┘ shape: (2, 3) ┌───────┬─────┬──────┐ │ index ┆ a ┆ b │ │ --- ┆ --- ┆ --- │ │ u32 ┆ i64 ┆ bool │ ╞═══════╪═════╪══════╡ │ 0 ┆ 1 ┆ true │ │ 2 ┆ 3 ┆ null │ └───────┴─────┴──────┘
Question: I am using pathlib.glob() and pathlib.rglob() to matching files from a directory and its subdirectories, respectively. Target files both are both lower case .txt and upper case .TXT files. According file paths were read from the filesystem as follows: import pathlib directory = pathlib.Path() files_to_create = ['a.txt', 'b.TXT'] suffixes_to_test = ['*.txt', '*.TXT'] for filename in files_to_create: filepath = directory / filename filepath.touch() for suffix in suffixes_to_test: files = [fp.relative_to(directory) for fp in directory.glob(suffix)] print(f'{suffix}: {files}')  The majority of the code base was developed on a Windows 10 machine (running Python 3.7.4) and was now moved to macOS Monterey 12.0.1 (running Python 3.10.1). On Windows both files a.txt and b.TXT are matching the patterns: *.txt: [WindowsPath('a.txt'), WindowsPath('b.TXT')] *.TXT: [WindowsPath('a.txt'), WindowsPath('b.TXT')]  In contrast, macOS only one file matches each pattern: *.txt: [PosixPath('a.txt')] *.TXT: [PosixPath('b.TXT')]  Therefore, I assume that the macOS file system might be case-sensitive, whereas the Windows one is not. According to Apple's User Guide the macOS file system used should not be case-sensitive by default but can be configured as such. Something similar might apply for Linux or Unix file systems as discussed here and here. Despite the reason for this differing behavior, I need to find a platform-agnostic way to get both capital TXT and lower case txt files. A rather naive workaround could be something like this: results = set([fp.relative_to(directory) for suffix in suffixes_to_test for fp in directory.glob(suffix)])  Which gives the desired output on both macOS and Windows: {PosixPath('b.TXT'), PosixPath('a.txt')}  However, is there a more elegant way? I could not find any option like ignore_case in pathlib's documentation.

Answer: What about something like: suffix = '*.[tT][xX][tT]' files = [fp.relative_to(directory) for fp in directory.glob(suffix)]  It is not so generalizable for a "case-insensitive glob", but it works well for limited and specific use-case like your glob of a specific extension.
Question: C++17 introduced the new structured bindings syntax: std::pair<int, int> p = {1, 2}; auto [a, b] = p;  Is there something similar in python3? I was thinking of using the "splat" operator to bind class variables to a list, which can be unpacked and assigned to multiple variables like such: class pair: def __init__(self, first, second): self.first = first self.second = second ... p = pair(1, 2) a, b = *p  Is this possible? And if so, how would I go by implementing this to work for my own classes? A tuple in Python works as a simple solution to this problem. However, built in types don't give much flexibility in implementing other class methods.

Answer: Yes, you can use __iter__ method since iterators can be unpacked too: class pair: def __init__(self, first, second): self.first = first self.second = second def __iter__(self): # Use tuple's iterator since it is the closest to our use case. return iter((self.first, self.second)) p = pair(1, 2) a, b = p print(a, b) # Prints 1 2
Question: I have two AWS-Lamdbda functions and I want Lambda A to determine a certain point in time like the 4. May 2022 10:00. Then I want Lambda B to be scheduled to run at this specific point in time. I'm probably able to achieve this by programmatically creating a AWS eventbrigde rule with Lambda A and use the cron pattern to match my point in time. Inside of Lambda B I would then need to delete that rule, because it's for a one time use. Can one of you think of a more elegant way to achieve this ? Thanks you for your wisdom ! Edit: My point in time is dynamic. I call a public API to find the point for Lambda B to run, so I can't use EventBridge directly. I plan on running Lambda A once a day to see if a new Lambda B run is necessary.

Answer: [Edit Nov 2022]: EventBridge Scheduler The new EventBridge Scheduler supports one-time schedules for events. The event will be invoked on the date and time you pass as the schedule expression: at(yyyy-mm-ddThh:mm:ss) in the boto3 EventScheduler client's create_schedule API.  Here are a few more options to schedule a Lambda run at an arbitrary point in time: Step Function A three-state Step Function orchestrates the two Lambdas. Lambda A obtains and outputs a timestamp. A Wait State waits until the timestamp passes. Then Lambda B runs. This approach is precise. N.B. Standard Workflow executions have a duration up to one year, so they can accomodate long waits. DynamoDB Streams + TTL Create a DynamoDB table with a TTL field and Dynamo DB Streams enabled. Set Lambda B as the Streams processor. Lambda A writes a record to the table with the timestamp as TTL. Shortly after the TTL timestamp passes, DynamoDB will trigger Lambda B. This approach won't give you the precision of the first approach, but will be cheaper if you have loads of events.
Question: When I tried to install libraries using pip install, sometimes this error message come up. ERROR: THESE PACKAGES DO NOT MATCH THE HASHES FROM THE REQUIREMENTS FILE. If you have updated the package versions, please update the hashes. Otherwise, examine the package contents carefully; someone may have tampered with them.  This error comes up when I am trying to build multiple images using docker-compose V2. What I have done:  pip install --no-cache-dir -r requirements.txt upgrading pip trying the old version of pip (20.0.2). change the version of the affected package. changing the dns  However, it still comes up randomly. The libraries that are referred to the error message also keep changing. Does anyone know the reason for this issue?

Answer: So I had the same issue, tried deleting the pip cache file and using the "--no-cache-dir" argument. None of those worked. I then came across a post that said they were experiencing this error because of a networking issue. So I switched off my VPN and everything worked perfectly! Not sure why this works, but it got the job done
Question: I have a program that iterates over several thousand PNG files on an SMB shared network drive (a 2TB Samsung 970 Evo+) and adds up their individual file sizes. Unfortunately, it is very slow. After profiling the code, it turns out 90% of the execution time is spent on one function: filesize += os.path.getsize(png) where each png variable is the filepath to a single PNG file (of the several thousands) in a for loop that iterates over each one obtained from glob.glob() (which, to compare, is responsible for 7.5% of the execution time).  The code can be found here: https://pastebin.com/SsDCFHLX Clearly there is something about obtaining the filesize over the network that is extremely slow, but I'm not sure what. Is there any way I can improve the performance? It takes just as long using filesize += os.stat(png).st_size too. When the PNG files are stored on the computer locally, the speed is not an issue. It specifically becomes a problem when the files are stored on another machine that I access over the local network with a gigabit ethernet cable. Both are running Windows 10. [2022-08-21 Update] I tried it again with a 10 gigabit network connection this time and noticed something interesting. The very first time I run the code on the network share, the profiler looks like this:  but if I run it again afterward, glob() takes up significantly less time while getsize() is about the same:  if I instead run this code with the PNG files stored on a local NVMe drive (WD SN750) rather than a newtwork drive, here's what the profiler looks like:  It seems like once it is run for a second time on the network share, something has gotten cached that allows glob() to run much faster on the network share, at around the same speed it would run at on the local NVMe drive. But getsize() remains extremely slow, about 1/10th of the speed as when local. Can somebody help me understand these two points:  Why is getsize() so much slower on the network share? Is there something that can be done to speed it up? Why is glob() slow the first time on the network share but not when I run it again immediately afterward? 

Answer: I don't know why getsize() is as slow as it is over the network, however to speed it up you could try calling it concurrently: import os from multiprocessing.pool import ThreadPool def get_total_filesize_concurrently(paths): total = 0 with ThreadPool(10) as pool: for size in pool.imap_unordered(lambda path: os.path.getsize(path), paths): total += size return total print(get_total_filesize_concurrently([ "E:\Path\To\File.txt", "E:\Path\To\File2.txt" "E:\Path\To\File3.txt" ... ]))  You can also play around with the number of threads defined in ThreadPool(10) to potentially increase performance even further.
Question: It seems they both behave exactly the same – both are like dicts but with . literal to access an item, however none of it is even a subclass of another from argparse import Namespace from types import SimpleNamespace issubclass(Namespace, SimpleNamespace) # False issubclass(SimpleNamespace, Namespace) # False  So, are there any differences between them two? Can argparse.Namespace be used in all cases?

Answer: There once was a proposal to have the argparse inherit. That said, just use the types one if you are using a new enough Python; that's what it is there for.
Question: I was training the model and saved it, now I am trying to load but unable to do. I have seen in previous post as well, but some reference links are not working or some things I tried, still not able to solve the problem. Code snippet: #load model with tf.io.gfile.GFile(args.model, "rb") as f: graph_def = tf.compat.v1.GraphDef() graph_def.ParseFromString(f.read()) # with tf.Graph().as_default() as graph: generated_image_1, generated_image_2, generated_image_3, = tf.graph_util.import_graph_def( graph_def, input_map={'input_image' : input_tensor, 'short_edge_1' : short_edge_1, 'short_edge_2' : short_edge_2, 'short_edge_3' : short_edge_3}, return_elements=['style_subnet/conv-block/resize_conv_1/output:0', 'enhance_subnet/resize_conv_1/output:0', 'refine_subnet/resize_conv_1/output:0'], producer_op_list=None )  Error Traceback (most recent call last): File "stylize.py", line 97, in <module> main() File "stylize.py", line 57, in main graph_def.ParseFromString(f.read()) google.protobuf.message.DecodeError: Error parsing message with type 'tensorflow.GraphDef'  Note: If need more information about this, will sure post add it here. Let me know

Answer: BG: I was getting errors while testing the code. In my case, it was solved with the help of freeze.py and a few modifications in the training file. And I found some other useful links while searching query. Link 1 Link 2
Question: since the last update to Mac OS Monterey 12.3 I get the following error message when compiling my Arduino sketch: exec: "python": executable file not found in $PATH  Unfortunately, I have not yet been able to find out how to solve this problem. I would be very grateful for ideas and suggestions.

Answer: Problem In MacOS 12.3 Apple removed python2.7 (python) from MacOS. Solution What I did to solve this is link python3 to python, I wouldn't recommend it because it's sus, I would recommend you wait until Arduino IDE fixes this issue in a later build. For the time being, you could try their Web IDE: Arduino Editor However, here are the instructions to link python3 to python:  If you don't have python3 installed, install it here in the link below: Python Install Page  Find your path for the current version of python3 you're using which python3 it'll show up with something like this: /Library/Frameworks/Python.framework/Versions/3.10/bin/python3  Copy that and use it to run this command that links python 3 to python. Replace the first file path with where your python3 is. ln -s -f INSERT_PATH_OF_PYTHON3 /usr/local/bin/python for example: ln -s -f /Library/Frameworks/Python.framework/Versions/3.10/bin/python3 /usr/local/bin/python
Question: PyCharm has a feature that colors your comments yellow if they include TODO or FIXME keywords. What is the way to add more keywords to the list and change the colors based on the keyword? Example: 

Answer: In PyCharm:  Press CTRL+ALT+S or navigate to Preferences/Settings Search TODO or go to Editor/TODO Add a pattern using the + button. For example, the pattern for ERROR keyword would be \berror\b.*. It can also be any other regex pattern. \b - word boundaries; .* - zero or more characters. Unselect Use color scheme TODO default colors and change the foreground color (ex. C00000 - red) Press OK and Apply  Results:  Settings:  Example ERROR pattern (red):
Question: sdf = df.to_sparse() has been deprecated. What's the updated way to convert to a sparse DataFrame?

Answer: These are the updated sparse conversions in pandas 1.0.0+.  How to convert dense to sparse Use DataFrame.astype() with the appropriate SparseDtype() (e.g., int): >>> df = pd.DataFrame({'A': [1, 0, 0, 0, 1, 0]}) >>> df.dtypes # A int64 # dtype: object >>> sdf = df.astype(pd.SparseDtype(int, fill_value=0)) >>> sdf.dtypes # A Sparse[int64, 0] # dtype: object  Or use the string alias for brevity: >>> sdf = df.astype('Sparse[int64, 0]')   How to convert sparse to dense Use DataFrame.sparse.to_dense(): >>> from scipy import sparse >>> sdf = pd.DataFrame.sparse.from_spmatrix(sparse.eye(3), columns=list('ABC')) >>> sdf.dtypes # A Sparse[float64, 0] # B Sparse[float64, 0] # C Sparse[float64, 0] # dtype: object >>> df = sdf.sparse.to_dense() >>> df.dtypes # A float64 # B float64 # C float64 # dtype: object   How to convert sparse to COO Use DataFrame.sparse.to_coo(): >>> from scipy import sparse >>> sdf = pd.DataFrame.sparse.from_spmatrix(sparse.eye(3), columns=list('ABC')) >>> sdf.dtypes # A Sparse[float64, 0] # B Sparse[float64, 0] # C Sparse[float64, 0] # dtype: object >>> df = sdf.sparse.to_coo() # <3x3 sparse matrix of type '<class 'numpy.float64'>' # with 3 stored elements in COOrdinate format> # (0, 0) 1.0 # (1, 1) 1.0 # (2, 2) 1.0