eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I have a number of coordinates (roughly 20000) for which I need to extract data from a number of NetCDF files each comes roughly with 30000 timesteps (future climate scenarios). Using the solution here is not efficient and the reason is the time spent at each i,j to convert "dsloc" to "dataframe" (look at the code below). ** an example NetCDF file could be download from here ** import pandas as pd import xarray as xr import time #Generate some coordinates coords_data = [{'lat': 68.04, 'lon': 15.20, 'stid':1}, {'lat':67.96, 'lon': 14.95, 'stid': 2}] crd= pd.DataFrame(coords_data) lat = crd["lat"] lon = crd["lon"] stid=crd["stid"] NC = xr.open_dataset(nc_file) point_list = zip(lat,lon,stid) start_time = time.time() for i,j,id in point_list: print(i,j) dsloc = NC.sel(lat=i,lon=j,method='nearest') print("--- %s seconds ---" % (time.time() - start_time)) DT=dsloc.to_dataframe() DT.insert(loc=0,column="station",value=id) DT.reset_index(inplace=True) temp=temp.append(DT,sort=True) print("--- %s seconds ---" % (time.time() - start_time))  which results is: 68.04 15.2 --- 0.005853414535522461 seconds --- --- 9.02660846710205 seconds --- 67.96 14.95 --- 9.028568267822266 seconds --- --- 16.429600715637207 seconds ---  which means each i,j takes around 9 seconds to process. Given lots of coordinates and netcdf files with large timesteps, I wonder if there a pythonic way that the code could be optimized. I could also use CDO and NCO operators but I found a similar issue using them too.

Answer: This is a perfect use case for xarray's advanced indexing using a DataArray index. # Make the index on your coordinates DataFrame the station ID, # then convert to a dataset. # This results in a Dataset with two DataArrays, lat and lon, each # of which are indexed by a single dimension, stid crd_ix = crd.set_index('stid').to_xarray() # now, select using the arrays, and the data will be re-oriented to have # the data only for the desired pixels, indexed by 'stid'. The # non-indexing coordinates lat and lon will be indexed by (stid) as well. NC.sel(lon=crd_ix.lon, lat=crd_ix.lat, method='nearest')  Other dimensions in the data will be ignored, so if your original data has dimensions (lat, lon, z, time) your new data would have dimensions (stid, z, time).
Question: I'm trying to figure out in what order this code runs: print( True in [True] in [True] ) False  even though: print( ( True in [True] ) in [True] ) True  and: print( True in ( [True] in [True] ) ) TypeError  If the first code is neither of these two last ones, then what?

Answer: in is comparing with chaining there so True in [True] in [True]  is equivalent to (except middle [True] is evaluated once) (True in [True]) and ([True] in [True])  which is True and False  which is False  This is all similar to 2 < 4 < 12  operation which is equivalent to (2 < 4) and (4 < 12).
Question: I'm trying to encrypt file that is larger than 1GB. I don't want to read it all to memory. I chose Fernet (cryptography.fernet) for this task, because it was most recommended (faster than asymetric solutions). I generated the key. Then I've created a script to encrypt:  key = Fernet(read_key()) with open(source, "rb") as src, open(destination, "wb") as dest: for chunk in iter(lambda: src.read(4096), b""): encrypted = key.encrypt(chunk) dest.write(encrypted)  and for decryption:  key = Fernet(read_key()) with open(source, "rb") as src, open(destination, "wb") as dest: for chunk in iter(lambda: src.read(4096), b""): decrypted = key.decrypt(chunk) dest.write(decrypted)  Encryption works - no surprise, but decryption is not. Firstly I thought that it might work, but it's not. I guess chunk size increases when encrypted, and then when I'm reading 4096 bytes, it's not a whole encrypted chunk. I've got an error trying to decrypt: Traceback (most recent call last): File "/redacted/path/venv/lib/python3.7/site-packages/cryptography/fernet.py", line 119, in _verify_signature h.verify(data[-32:]) File "/redacted/path/venv/lib/python3.7/site-packages/cryptography/hazmat/primitives/hmac.py", line 74, in verify ctx.verify(signature) File "/redacted/path/venv/lib/python3.7/site-packages/cryptography/hazmat/backends/openssl/hmac.py", line 75, in verify raise InvalidSignature("Signature did not match digest.") cryptography.exceptions.InvalidSignature: Signature did not match digest. During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/redacted/path/main.py", line 63, in <module> decrypted = key.decrypt(chunk) File "/redacted/path/venv/lib/python3.7/site-packages/cryptography/fernet.py", line 80, in decrypt return self._decrypt_data(data, timestamp, time_info) File "/redacted/path/venv/lib/python3.7/site-packages/cryptography/fernet.py", line 137, in _decrypt_data self._verify_signature(data) File "/redacted/path/venv/lib/python3.7/site-packages/cryptography/fernet.py", line 121, in _verify_signature raise InvalidToken cryptography.fernet.InvalidToken  Is there's a way to solve this? Maybe there's a better (simpler) approach with different solution than fernet?

Answer: Fernet is not supposed to be used in a streaming fashion. They explain that in the documentation: From the documentation (last section):  Limitations Fernet is ideal for encrypting data that easily fits in memory. As a design feature it does not expose unauthenticated bytes. This means that the complete message contents must be available in memory, making Fernet generally unsuitable for very large files at this time.
Question: I have to use the Process method with "spawn" context in Linux. Then I write a sample code as follows: from multiprocessing import Value import multiprocessing class Test(object): def __init__(self, m_val): print("step1") self.m_val = m_val print("step2") self.m_val_val = m_val.value self.prints() def prints(self): print("self.m_val_val:%d"%self.m_val_val) def main(m_val): t = Test(m_val) if __name__ == "__main__": N = 2 procs = [] v = Value("i",10) for i in range(0,N): proc_i = multiprocessing.get_context("spawn").Process(target=main,args=(v,)) proc_i.daemon=True procs.append(proc_i) for i in range(0,N): procs[i].start() for i in range(0,N): procs[i].join()  When I run this code in Linux, it will print: step1 step2 step1 step2  while in Windows, the print content will be: step1 step2 self.m_val_val:10 step1 step2 self.m_val_val:10  Besides, there is no error information printed on the screen. So, how can I solve this problem, i.e., how to use multiprocessing Value in among processes while using "spawn" context in Linux?

Answer: The problem is that you are creating Value in the default context, which is fork on Unix. You can resolve this by setting the default start context to "spawn": multiprocessing.set_start_method("spawn") # Add this v = Value("i",10)  Better yet, create the Value in the context explicitly: # v = Value("i",10) # Change this ctx = multiprocessing.get_context("spawn") # to this v = ctx.Value("i",10) # for i in range(0,N): # proc_i = multiprocessing.get_context("spawn").Process(target=main,args=(v,)) # (Optional) Refactor this proc_i = ctx.Process(target=main,args=(v,)) # to this  Reference From https://docs.python.org/3/library/multiprocessing.html#contexts-and-start-methods:  spawn Available on Unix and Windows. The default on Windows and macOS. fork Available on Unix only. The default on Unix.   Note that objects related to one context may not be compatible with processes for a different context. In particular, locks created using the fork context cannot be passed to processes started using the spawn or forkserver start methods.   Reproducing this issue on macOS This issue can be reproduced on macOS with Python < 3.8. From https://docs.python.org/3/library/multiprocessing.html#contexts-and-start-methods:  Changed in version 3.8: On macOS, the spawn start method is now the default.  To reproduce this issue on macOS with Python 3.8 and above: multiprocessing.set_start_method("fork") # Add this v = Value("i",10)  Error message and stack trace  OSError: [Errno 9] Bad file descriptor  Traceback (most recent call last): File "/path/to/python/multiprocessing/process.py", line 258, in _bootstrap self.run() File "/path/to/python/multiprocessing/process.py", line 93, in run self._target(*self._args, **self._kwargs) File "/path/to/file.py", line 18, in main t = Test(m_val) File "/path/to/file.py", line 10, in __init__ self.m_val_val = m_val.value File "<string>", line 3, in getvalue OSError: [Errno 9] Bad file descriptor
Question: I am managing dependencies in my Python project via Poetry. Now I want to run this project in a machine which is different from my dev machine. To install dependecies, I simply run this command from the root directory: $ poetry install  but then it raises the following errors: Updating dependencies Resolving dependencies... Writing lock file Package operations: 70 installs, 0 updates, 0 removals • Installing colorama (0.4.4) • Installing tzdata (2021.1) ValueError File \C:\Users\tteguayco\AppData\Local\pypoetry\Cache\artifacts\9e\b3\11\7d87ac44fdb2d557301f1f4086a37c080d1482a98751abe7cdbabbad26\colorama-0.4.4-py2.py3-none-any.whl does not exist at ~\AppData\Local\Programs\Python\Python39\lib\site-packages\poetry\core\packages\file_dependency.py:40 in __init__ 36│ except FileNotFoundError: 37│ raise ValueError("Directory {} does not exist".format(self._path)) 38│ 39│ if not self._full_path.exists(): → 40│ raise ValueError("File {} does not exist".format(self._path)) 41│ 42│ if self._full_path.is_dir(): 43│ raise ValueError("{} is a directory, expected a file".format(self._path)) 44│ ValueError File \C:\Users\tteguayco\AppData\Local\pypoetry\Cache\artifacts\45\2d\cb\6443e36999e7ab3926d5385dfac9ee9ea2a62f8111ff71abb6aff70674\tzdata-2021.1-py2.py3-none-any.whl does not exist at ~\AppData\Local\Programs\Python\Python39\lib\site-packages\poetry\core\packages\file_dependency.py:40 in __init__ 36│ except FileNotFoundError: 37│ raise ValueError("Directory {} does not exist".format(self._path)) 38│ 39│ if not self._full_path.exists(): → 40│ raise ValueError("File {} does not exist".format(self._path)) 41│ 42│ if self._full_path.is_dir(): 43│ raise ValueError("{} is a directory, expected a file".format(self._path)) 44│  It would be good to know what these *.whl are and how they are used by Poetry.

Answer: Specifically I found that deleting the AppData\Local\pypoetry\Cache\artifacts folder (I'm on Windows 10) worked for me. virtualenvs for other projects may be in AppData\Local\pypoetry\Cache\virtualenvs so you might not want to delete the root cache folder at AppData\Local\pypoetry\Cache in its entirety.
Question: Assume we have some function func that maps instances of class A to instances of class B, i.e. it has the signature Callable[[A], B]. I want to write a class decorator autofunc for subclasses of A that automatically applies func to instances as they are created. For example, think of automatic jit-compilation based on a global environment variable. This can be done via from functools import wraps def autofunc(basecls): @wraps(basecls, updated=()) class WrappedClass(basecls): def __new__(cls, *args, **kwargs): instance = basecls(*args, **kwargs) return func(instance) return WrappedClass  Then the following two are roughly equivalent:    class C(A):   ...    instance = func(C())   @autofunc  class C(A):   ...    instance = C()     In my naivety, I tried def autofunc(basecls: type[A]) -> type[B]: @wraps(basecls, updated=()) class WrappedClass(basecls): def __new__(cls, *args, **kwargs): instance = basecls(*args, **kwargs) return func(instance) return WrappedClass  which mypy really dislikes, raising erros:  error: Variable "basecls" is not valid as a type [valid-type] error: Invalid base class "basecls" [misc] also, there is the issue that WrapperClass.__new__ returns an instance of B and not of WrapperClass.  Is there any way to properly type hint such a class decorator currently, or is mypy not capable to work this yet?  Example code: from functools import wraps class A: pass class B: pass def func(cl: A) -> B: print(f"Replacing {cl=}") return B() def autofunc(basecls: type[A]) -> type[B]: @wraps(basecls, updated=()) class WrappedClass(basecls): def __new__(cls, *args, **kwargs): instance = basecls() return func(instance) return WrappedClass 

Answer: This is simply a bug in mypy: https://github.com/python/mypy/issues/5865
Question: I'm trying to run a script in AWS lambda and I want to output info level logs to the console after the script runs. I've tried looking for help from This post on using logs in lambda buy haven't had any success. I think AWS Cloudwatch is overriding my configuration shown bellow.  import logging # log configuration logging.basicConfig( format='%(levelname)s: %(message)s', level=logging.INFO, encoding="utf-8" )  I want to set the logging level to logging.info. How can I do this? The runtime is python 3.9

Answer: from my understanding, I think one fix would be to add this: logging.getLogger().setLevel('INFO')  I believe that logging.basicConfig(level=...) affects the minimum log level at which logs show up in the console, but across all loggers. The one above explicitly sets the minimum enabled level for the root logger, i.e. logging.getLogger(). The enabled level for each logger determines at what level messages will be logged - otherwise, each call to a logger method like logging.info is basically a no-op. So essentially, the basicConfig and setLevel calls are separate, but they work together to determine if a library's logs are printed to the console. For example, you can set basicConfig(level='DEBUG') so that the debug-level logs for all libraries should get printed out. But if you want to make an exception for one library like botocore for example, you can use logging.getLogger('botocore').setLevel('WARNING'), and that will set the minimum enabled level for that library to WARNING, so only messages logged by this library above that minimum level get printed out to the console.
Question: Consider the following example: import itertools import numpy as np a = np.arange(0,5) b = np.arange(0,3) c = np.arange(0,7) prods = itertools.product(a,b,c) for p in prods: print(p)  This iterate over the products in the following order: (0, 0, 0) (0, 0, 1) (0, 0, 2) (0, 0, 3) (0, 0, 4) (0, 1, 0)  But I would much rather have the products given in order of their sum, e.g. (0, 0, 0) (0, 0, 1) (0, 1, 0) (1, 0, 0) (0, 1, 1) (1, 0, 1) (1, 1, 0) (0, 0, 2)  How can I achieve this without storing all combinations in memory? Note: a b and c are always ranges, but not necessarily with the same maximum. There is also no 2nd-level ordering when the sums of two products are equal, i.e. (0,1,1) is equivalent to (2,0,0).

Answer: The easiest way to do this without storing extra products in memory is with recursion. Instead of range(a,b), pass in a list of (a,b) pairs and do the iteration yourself: def prod_by_sum(range_bounds: List[Tuple[int, int]]): """ Yield from the Cartesian product of input ranges, produced in order of sum. >>> range_bounds = [(2, 4), (3, 6), (0, 2)] >>> for prod in prod_by_sum(range_bounds): ... print(prod) (2, 3, 0) (2, 3, 1) (2, 4, 0) (3, 3, 0) (2, 4, 1) (2, 5, 0) (3, 3, 1) (3, 4, 0) (2, 5, 1) (3, 4, 1) (3, 5, 0) (3, 5, 1) """ def prod_by_sum_helper(start: int, goal_sum: int): low, high = range_bounds[start] if start == len(range_bounds) - 1: if low <= goal_sum < high: yield (goal_sum,) return for current in range(low, min(high, goal_sum + 1)): yield from ((current,) + extra for extra in prod_by_sum_helper(start + 1, goal_sum - current)) lowest_sum = sum(lo for lo, hi in range_bounds) highest_sum = sum(hi - 1 for lo, hi in range_bounds) for goal_sum in range(lowest_sum, highest_sum + 1): yield from prod_by_sum_helper(0, goal_sum)  which has output for range_bounds = [(0, 5), (0, 3), (0, 7)] starting with: (0, 0, 0) (0, 0, 1) (0, 1, 0) (1, 0, 0) (0, 0, 2) (0, 1, 1) (0, 2, 0) (1, 0, 1) (1, 1, 0) (2, 0, 0)  You can do this exact process iteratively by modifying a single list and yielding copies of it, but the code either becomes more complicated or less efficient. You can also trivially modify this to support steps besides 1, however that does work less efficiently with larger and larger steps since the last range might not contain the element needed to produce the current sum. That seems unavoidable, because at that point you'd need to solve a difficult computational problem to efficiently loop over those products by sum.
Question: I am confused about the use of type hints and docstrings. Aren't they duplicate information? For example: def my_func(name: str): """ print a name. Parameters ---------- name : str a given name """ print(name)  Isn't the information name: str given twice?

Answer: Putting the type as a type hint and as part of the docstring would be redundant. It is also prone to human errors since one could easily forget updating one of them, thus effort is constantly needed to keep them both in sync. The documentation for type hints also mentions it:  Docstrings. There is an existing convention for docstrings, based on the Sphinx notation (:type arg1: description). This is pretty verbose (an extra line per parameter), and not very elegant. We could also make up something new, but the annotation syntax is hard to beat (because it was designed for this very purpose).  But all of this depends on your usecase.  If you are using a static-type checker such as MyPy (e.g. it would fail if you passed an int 123 to a variable with type hint str) or an IDE such as PyCharm (e.g. it would highlight inconsistencies between type hint and passed arguments), then you should keep on using type hints. This is the preferable way, as it points out possible errors in the code you've written. If you are using tools such as Sphinx, or Swagger, or something else, note that these tools display your classes and methods along with their docstrings for the purposes of documentation. So if you want to maintain such documentation for clarity to other readers and want to have the types included, then you might want to put the type hint in the docstring as well. But if you think such detail isn't needed as most of the time only the description in the docstring is relevant, then there is no need to add the type hint in the docstring.  In the long run, the more sustainable way is to have those tools (Sphinx, Swagger, etc.) use the type hint as part of the documentation instead of relying solely on the text in the docstring. For sphinx, I found this library sphinx-autodoc-typehints that performs it already.  Allowing you to migrate from this: def format_unit(value, unit): """ Formats the given value as a human readable string using the given units. :param float|int value: a numeric value :param str unit: the unit for the value (kg, m, etc.) :rtype: str """ return '{} {}'.format(value, unit)  to this: from typing import Union def format_unit(value: Union[float, int], unit: str) -> str: """ Formats the given value as a human readable string using the given units. :param value: a numeric value :param unit: the unit for the value (kg, m, etc.) """ return '{} {}'.format(value, unit)   So, it seems like there are already enhancements on this topic that will make the definition of types more consistent and less redundant.
Question: I have a function of the following structure: def get_something_from_data(data: Mapping[str, str], default: Optional[str] = None) -> Optional[str]: """Get something out of `data` if it is there, if not return the value of `default`. If `default` is not provided, return None. """  So this can return either Optional[str] if default is omitted, or will always return str if default has an str value. Now I call this in code similar to: has_value = get_something_from_data(data, "fallback") return has_value.endswith("k")  Even though in this case has_value will always be an str, running mypy on this code generates an error because it thinks has_value might be None and will not have endswith. So I tried typing my function differently: DT = TypeVar('DT', str, None) def get_something_from_data(data: Mapping[str, str], default: DT = None) -> Union[str, DT]: pass  This works, but now when I call mypy on code like: has_value = get_something_from_data(data) return has_value.endswith("k") # might fail as has_value can be None  mypy doesn't throw an error, even though the code is potentially dangerous. Is there a way to properly type this function so mypy generates errors based on whether or not a default value was provided?

Answer: Using typing.overload you can describe multiple combinations of arguments and return types for a function from typing import overload, Mapping, Optional @overload def get_something_from_data(data: Mapping[str, str], default: None = None) -> Optional[str]: ... @overload def get_something_from_data(data: Mapping[str, str], default: str) -> str: ... def get_something_from_data(data, default=None): pass
Question: I have a question about adding delay after calling various functions. Let's say I've function like: def my_func1(): print("Function 1") def my_func2(): print("Function 2") def my_func3(): print("Function 3")  Currently I've added delay between invoking them like below: delay = 1 my_func1() time.sleep(delay) my_func2() time.sleep(delay) my_func3() time.sleep(delay)  As you can see I needed a few times time.sleep, which I would like to avoid. Using decorator is also not an option, since it might be that I would like to avoid delay when calling one of this function not in a group. Do you have any tip how to beautify this?

Answer: I've tested this based on "How to Make Decorators Optionally Turn On Or Off" (How to Make Decorators Optionally Turn On Or Off) from time import sleep def funcdelay(func): def inner(): func() print('inner') sleep(1) inner.nodelay = func return inner @funcdelay def my_func1(): print("Function 1") @funcdelay def my_func2(): print("Function 2") @funcdelay def my_func3(): print("Function 3") my_func1() my_func2() my_func3() my_func1.nodelay() my_func2.nodelay() my_func3.nodelay()  Output: Function 1 inner Function 2 inner Function 3 inner Function 1 Function 2 Function 3  You can see that it can bypass the delay.