eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I have a nested dict with boolean values, like: assignments_dict = {"first": {'0': True, '1': True}, "second": {'0': True, '1': False}, }  and an array, with a number of elements equal to the number of True values in the assignments_dict: results_array = [10, 11, 12]  and, finally, a dict for results structured this way: results_dict = {"first": {'0': {'output': None}, '1': {'output': None}}, "second": {'0': {'output': None}, '1': {'output': None}}, }  I need to go through the fields in assignment_dict, check if they are True, and if they are take the next element of results_array and substitute it to the corresponding field in results_dict. So, my final output should be: results_dict = {'first': {'0': {'output': 10}, '1': {'output': 11}}, 'second': {'0': {'output': 12}, '1': {'output': None}}}  I did it in a very simple way: # counter used to track position in array_outputs counter = 0 for outer_key in assignments_dict: for inner_key in assignments_dict[outer_key]: # check if every field in assignments_dict is True/false if assignments_dict[outer_key][inner_key]: results_dict[outer_key][inner_key]["output"] = results_array[counter] # move on to next element in array_outputs counter += 1  but I was wondering if there's a more pythonic way to solve this.

Answer: results_iter = iter(results_array) for key, value in assignments_dict.items(): for inner_key, inner_value in value.items(): if inner_value: results_dict[key][inner_key]['output'] = next(results_iter) print(results_dict)  Output: {'first': {'0': {'output': 10}, '1': {'output': 11}}, 'second': {'0': {'output': 12}, '1': {'output': None}}}
Question: I'm looking at a setup.py with this syntax: from setuptools import setup setup( ... tests_require=["h5py>=2.9=mpi*", "mpi4py"] )  I understand the idea of the ">=" where h5py should be at least version 2.9, but I cannot for the life of me understand the =mpi* afterwards. Is it saying the version should somehow match the mpi version, while also being at least 2.9? I can't find anything that explains specifying python package versions that also explains the use of a single =. The only other place I've found it used is some obscure blog post that seemed to imply it was sort of like importing the package with an alias, which doesn't make much sense to me; and also the mpi4py docs that include a command line snippet conda install -c conda-forge h5py=*=mpi* netcdf4=*=mpi* but don't really explain it.

Answer: Short answer The =mpi* qualifier says that you want to install h5py pre-compiled with MPI support. Details If you look at the documentation for h5py, you'll see references to having to build it with or without MPI explicitly (e.g., see https://docs.h5py.org/en/latest/build.html). When you look at the conda-forge download files (https://anaconda.org/conda-forge/h5py/files) you'll also see that there are a bunch of nompi variants and a bunch of mpi variants. Adding =mpi* triggers getting a version that's been compiled with MPI so that you get parallel MPI support, while I suspect the default version would come without MPI support. Experimentation with and without When I do conda install -c conda-forge h5py=3.7, conda proposes to download this bundle: h5py-3.7.0-nompi_py39hd4deaf1_100  But when I did conda install =c conda-forget h5py=3.7=mpi*, I expected a ...-mpi_py... bundle but instead it just failed because I'm on Windows and MPI is not supported on Windows as far as I can tell. (And that makes sense, HPC clusters with MPI run on Linux.)
Question: Forgive my ignorance.. I'm trying to learn how to schedule python scripts with Google Cloud. After a bit of research, I've seen many people suggest Docker + Google Cloud Run + Cloud Scheduler. I've attempted to get a "hello world" example working, to no avail. Code hello.py print("hello world")  Dockerfile # For more information, please refer to https://aka.ms/vscode-docker-python FROM python:3.8-slim # Keeps Python from generating .pyc files in the container ENV PYTHONDONTWRITEBYTECODE=1 # Turns off buffering for easier container logging ENV PYTHONUNBUFFERED=1 WORKDIR /app COPY . /app # Creates a non-root user with an explicit UID and adds permission to access the /app folder # For more info, please refer to https://aka.ms/vscode-docker-python-configure-containers RUN adduser -u 5678 --disabled-password --gecos "" appuser && chown -R appuser /app USER appuser # During debugging, this entry point will be overridden. For more information, please refer to https://aka.ms/vscode-docker-python-debug CMD ["python", "hello.py"]  Steps  Create a repo with Google Cloud Artifact Registry gcloud artifacts repositories create test-repo --repository-format=docker \ --location=us-central1 --description="My test repo"   Build the image docker image build --pull --file Dockerfile --tag 'testdocker:latest' .   Configure auth gcloud auth configure-docker us-central1-docker.pkg.dev   Tag the image with a registry name docker tag testdocker:latest \ us-central1-docker.pkg.dev/gormanalysis/test-repo/testdocker:latest   Push the image to Artifact Registry docker push us-central1-docker.pkg.dev/gormanalysis/test-repo/testdocker:latest    Deploy to Google Cloud Run      Error At this point, I get the error  The user-provided container failed to start and listen on the port defined provided by the PORT=8080 environment variable.  I've seen posts like this which say to add app.run(port=int(os.environ.get("PORT", 8080)),host='0.0.0.0',debug=True)  but this looks like a flask thing, and my script doesn't use flask. I feel like i have a fundamental misunderstanding of how this is supposed to work. Any help would be appreciated it.

Answer: UPDATE I've documented my problem and solution in much more detail here »  I had been trying to deploy my script as a Cloud Run Service. I should've tried deploying it as a Cloud Run Job. The difference is that cloud run services require your script to listen for a port. jobs do not.  Confusingly, you cannot deploy a cloud run job directly from Artifact Registry. You have to start from the cloud run dashboard.
Question: I've used the default pylint from git actions to check my project for any errors. There are some errors that I want to ignore though. If it was in vscode you could ignore them in settings.json. How do I ignore them in git actions?  name: Pylint on: [push] jobs: build: runs-on: ubuntu-latest strategy: matrix: python-version: ["3.10"] steps: - uses: actions/checkout@v3 - name: Set up Python ${{ matrix.python-version }} uses: actions/setup-python@v3 with: python-version: ${{ matrix.python-version }} - name: Install dependencies run: | python -m pip install --upgrade pip pip install pylint - name: Analysing the code with pylint run: | pylint $(git ls-files '*.py') 

Answer:  you could use noqa comments for ignore specific lines you can run probably pylint with option disable or something like that:  pylint -d C0114,C0116 $(git ls-files '*.py')  this would disable warnings with the codes C0114 and C0116
Question: Let's say we have a fixture that allocated some unmanaged resources and releases them like in the following example: @pytest.fixture def resource(): res = driver.Instance() yield res res.close()  Is there a guarantee that the resource will be released even if something bad happens during the test that utilizes that fixture? If there is no such guarantee, maybe the following pattern will be better? @pytest.fixture def resource(q): res = driver.Instance() def finalize(): res.close() q.addfinalizer(finalize()) return res 

Answer: It depends on what happens before yield:  If a yield fixture raises an exception before yielding, pytest won’t try to run the teardown code after that yield fixture’s yield statement. But, for every fixture that has already run successfully for that test, pytest will still attempt to tear them down as it normally would.  You say, "even if something bad happens during the test that utilizes that fixture", which implies that the yield has executed. As long as the fixture yields, pytest will attempt to perform the teardown. I don't think manually finalizing offers any further guarantees:  While yield fixtures are considered to be the cleaner and more straightforward option, there is another choice, and that is to add “finalizer” functions directly to the test’s request-context object. It brings a similar result as yield fixtures, but requires a bit more verbosity.  The section on safe fixtures offers some good tips on how to handle teardown code safely, nicely summarized by this line:  The safest and simplest fixture structure requires limiting fixtures to only making one state-changing action each, and then bundling them together with their teardown code
Question: I retrieve timezones from airports of the world, using TimezoneFinder().timezone_at applied on longitude and latitude of airports. And when I want to use these timezones (to compute times of departure ad arrival of flights) everything works except America/Ciudad_Juarez. This simple code: from timezonefinder import TimezoneFinder from pytz import timezone tz = TimezoneFinder().timezone_at(lng=-106.48333, lat=31.73333) # retrieves 'America/Ciudad_Juarez' timezone(tz)  Generates this error: UnknownTimeZoneError: 'America/Ciudad_Juarez'  I checked in this excellent wikipedia page, this timezone is correct, canonical they say. I am surprised that a timezone obtained through timezonefinder is not recognised by pytz. How can I cleanly solve this?

Answer: This timezone was only very recently added, as Ciudad Juárez decided to align it's time with the US for DST. See http://mm.icann.org/pipermail/tz-announce/2022-November/000076.html for the announcement. It looks like pytz has not been updated to include that change just yet.
Question: I have a python library packaged using poetry. I have the following requirements Inside the poetry project # example_library.py def get_version() -> str: # return the project version dynamically. Only used within the library def get_some_dict() -> dict: # meant to be exported and used by others return { "version": get_version(), "data": #... some data }  In the host project, I want the following test case to pass no matter which version of example_library I'm using from example_library import get_some_dict import importlib.metadata version = importlib.metadata.version('example_library') assert get_some_dict()["version"] == version  I have researched ideas about reading the pyproject.toml file but I'm not sure how to make the function read the toml file regardless of the library's location. The Poetry API doesn't really help either, because I just want to read the top level TOML file from within the library and get the version number, not create one from scratch.

Answer: Use importlib.metadata.version() from Python's own standard library: import importlib.metadata version = importlib.metadata.version('ProjectName')  This is not specific to Poetry and will work for any project (library or application) whose metadata is readable. Make sure that ProjectName is installed. Where ProjectName can be your own library or a 3rd party library, it does not matter. And note that a so-called "editable" installation is also good enough. Documentation for importlib.metadata.version(). See also:  https://stackoverflow.com/a/65594622
Question: t.py value = 0 def change_value(): global value value = 10  s.py import t from t import value t.change_value() print(f'test1: {t.value}') print (f'test2: {value}')  Output  test1: 10 test2: 0  Why isn't it not returning the changed value in the test2 ?

Answer: This is how import works in Python. You should read what the documentation says on the import statement carefully. Specifically, when you use the from import syntax on a module attribute, Python looks up the value of that attribute, and then "a reference to that value is stored in the local namespace". So, if value is 10, then after from t import value you have a reference to 10. That 10 stays a 10 no matter what happens to t.value. Just to drive home the point about reference semantics vs. value semantics, consider what would happen if you had this in t.py instead: value = [] def change_value(): global value # not actually needed anymore value.append(10) def change_reference(): global value value = value + [10]  Contrast the differences between calling t.change_value() and t.change_reference(). In the first case, both prints would be the same, while in the second, they would be different. This is because after calling change_value() we only have one list, and the name value in s.py is referring to it. With change_reference() we have two lists, the original (empty) one and a new one, and value in s.py is still referring to the original one.
Question: I am using a string that uses the following characters: 0-9 a-f A-F - >  The mixture of the greater than and hyphen must be: -> -->  Here is the regex that I have so far: [0-9a-fA-F\-\>]+  I tried these others using exclusion with ^ but they didn't work: [^g-zG-Z][0-9a-fA-F\-\>]+ ^g-zG-Z[0-9a-fA-F\-\>]+ [0-9a-fA-F\-\>]^g-zG-Z+ [0-9a-fA-F\-\>]+^g-zG-Z [0-9a-fA-F\-\>]+[^g-zG-Z]  Here are some samples: "0912adbd->12d1829-->218990d" "ab2c8d-->82a921->193acd7" 

Answer: Firstly, you don't need to escape - and > Here's the regex that worked for me: ^([0-9a-fA-F]*(->)*(-->)*)*$  Here's an alternative regex: ^([0-9a-fA-F]*(-+>)*)*$ What does the regex do?  ^ matches the beginning of the string and $ matches the ending. * matches 0 or more instances of the preceding token Created a big () capturing group to match any token. [0-9a-fA-F] matches any character that is in the range. (->) and (-->) match only those given instances.  Putting it into a code: import re regex = "^([0-9a-fA-F]*(->)*(-->)*)*$" re.match(re.compile(regex),"0912adbd->12d1829-->218990d") re.match(re.compile(regex),"ab2c8d-->82a921->193acd7") re.match(re.compile(regex),"this-failed->so-->bad")  You can also convert it into a boolean: print(bool(re.match(re.compile(regex),"0912adbd->12d1829-->218990d"))) print(bool(re.match(re.compile(regex),"ab2c8d-->82a921->193acd7"))) print(bool(re.match(re.compile(regex),"this-failed->so-->bad")))  Output: True True False  I recommend using regexr.com to check your regex.
Question: In order to understand how threads work in Python, I wrote the following simple function: def sum_list(thelist:list, start:int, end:int): s = 0 for i in range(start,end): s += thelist[i]**3//10 return s  Then I created a list and tested how much time it takes to compute its sum: LISTSIZE = 5000000 big_list = list(range(LISTSIZE)) start = time.perf_counter() big_sum=sum_list(big_list, 0, LISTSIZE) print(f"One thread: sum={big_sum}, time={time.perf_counter()-start} sec")  It took about 2 seconds. Then I tried to partition the computation into threads, such that each thread computes the function on a subset of the list: THREADCOUNT=4 SUBLISTSIZE = LISTSIZE//THREADCOUNT start = time.perf_counter() with concurrent.futures.ThreadPoolExecutor(THREADCOUNT) as executor: futures = [executor.submit(sum_list, big_list, i*SUBLISTSIZE, (i+1)*SUBLISTSIZE) for i in range(THREADCOUNT)] big_sum = 0 for res in concurrent.futures.as_completed(futures): # return each result as soon as it is completed: big_sum += res.result() print(f"{THREADCOUNT} threads: sum={big_sum}, time={time.perf_counter()-start} sec")  Since I have a 4-cores CPU, I expected it to run 4 times faster. But it did not: it ran in about 1.8 seconds on my Ubuntu machine (on my Windows machine, with 8 cores, it ran even slower than the single-thread version: about 2.2 seconds). Is there a way to use ThreadPoolExecutor (or another threads-based mechanism in Python) so that I can compute this function faster?

Answer: The problem is that the function you are trying to make faster is CPU-bound and the Python Global Interpreter Lock (GIL) prevents any performance gain from parallelisation of such code. In Python, threads are wrapper around genuine OS thread. However, in order to avoid race conditions due to concurrent execution, only one thread can access the Python interpreter to execute bytecode at a time. This restriction is enforced by a lock called the GIL. Thus in Python, true multithreading cannot be achieved and multiprocessing should be used instead. However, note that the GIL is not locked by IO operations (file reading, networking, etc.) and some library code (numpy, etc.) so these operations can still benefit from Python multithreading. The function sum_list used neither of those operations so it will not benefit from Python multithreading. You can use ProcessPoolExecutor to effectively get parallelism but this may copy the input list in your case. Multiprocessing is equivalent to launching multiple independent Python interpreters, thus the GIL's (one per intepreter) is not an issue anymore. However, multiprocessing incurs performance penalties during inter-process communication.
Question: According to this documentation for Vue's SSR, it is possible to use node.js to render an app and return it using an express server. Is is possible to do the same with FastAPI? Or is using Jinja2 templates or SPA the only solution? Problems:  No SPA: To help with SEO No SSG: Too many pages will be generated. Some need to be generated dynamically. No Jinja2/Python Templates: Node modules aren't built, bundled and served. All modules have to served from a remote package CDN.  I have a feeling that maybe changing the Vue 3 delimiters and then building the project and serving the files as Jinja2 templates is the solution, but I'm not sure how it would work with Vue's routers. I know the /dist folder can be served on the default route and then use a catchall can be used to display files that do exist. Possible Solution @app.get("/", response_class=FileResponse) def read_index(request: Request): index = f"{static_folder}/index.html" return FileResponse(index) @app.get("/{catchall:path}", response_class=FileResponse) def read_index(request: Request): path = request.path_params["catchall"] file = static_folder + path if os.path.exists(file): return FileResponse(file) index = f"{static_folder}/index.html" return FileResponse(index)  Questions  If there is a way to do SSR with FastAPI and Vue 3, what is it? If there is no direct way, how do I combine Vue's built /dist with Jinja2 templates to serve dynamic pages? 

Answer: There are several options available, such as Nuxt.js, Quasar, and Gridsome, which provide support for SSR with FastAPI and Vue 3.
Question: I am looking for guidance/best practice to approach a task. I want to use Azure-Databricks and PySpark. Task: Load and prepare data so that it can be efficiently/quickly analyzed in the future. The analysis will involve summary statistics, exploratory data analysis and maybe simple ML (regression). Analysis part is not clearly defined yet, so my solution needs flexibility in this area. Data: session level data (12TB) stored in 100 000 single line JSON files. JSON schema is nested, includes arrays. JSON schema is not uniform but new fields are added over time - data is a time-series. Overall, the task is to build an infrastructure so the data can be processed efficiently in the future. There will be no new data coming in. My initial plan was to:  Load data into blob storage  Process data using PySpark  flatten by reading into data frame save as parquet (alternatives?)   Store in a DB so the data can be quickly queried and analyzed  I am not sure which Azure solution (DB) would work here Can I skip this step when data is stored in efficient format (e.g. parquet)?   Analyze the data using PySpark by querying it from DB (or from blob storage when in parquet)   Does this sound reasonable? Does anyone has materials/tutorials that follow similar process so I could use them as blueprints for my pipeline?

Answer: Yes, it's sound reasonable, and in fact it's quite standard architecture (often referred as lakehouse). Usual implementation approach is following:  JSON data loaded into blob storage are consumed using Databricks Auto Loader that provides efficient way of ingesting only new data (since previous run). You can trigger pipeline regularly, for example, nightly, or run it continuously if data arriving all the time. Auto Loader is also handling schema evolution of input data.  Processed data is better to store as Delta Lake tables that provide better performance than "plain" Parquet due use of additional information in the transaction log so it's possible to efficiently access only necessary data. (Delta Lake is built on top of Parquet, but has more capabilities).  Processed data then could be accessed via Spark code, or via Databricks SQL (it could be more efficient for reporting, etc., as it's heavily optimized for BI workloads). Due the big amount of data, storing them in some "traditional" database may not be very efficient or be very costly.   P.S. I would recommend to look on implementing this with Delta Live Tables that may simplify development of your pipelines. Also, you may have access to Databricks Academy that has introductory courses about lakehouse architecture and data engineering patterns. If you don't have access to it, you can at least look to Databricks courses published on GitHub.
Question: I've have dataframe with column b with list elements, I need to create column c that counts number elements in list for every row. Here is toy example in Pandas: import pandas as pd df = pd.DataFrame({'a': [1,2,3], 'b':[[1,2,3], [2], [5,0]]}) a b 0 1 [1, 2, 3] 1 2 [2] 2 3 [5, 0] df.assign(c=df['b'].str.len()) a b c 0 1 [1, 2, 3] 3 1 2 [2] 1 2 3 [5, 0] 2  Here is my equivalent in Polars: import polars as pl dfp = pl.DataFrame({'a': [1,2,3], 'b':[[1,2,3], [2], [5,0]]}) dfp.with_columns(pl.col('b').map_elements(lambda x: len(x)).alias('c'))  I've a feeling that .map_elements(lambda x: len(x)) is not optimal. Is a better way to do it in Polars?

Answer: You can use .list.len() df.with_columns(c = pl.col("b").list.len())  shape: (3, 3) ┌─────┬───────────┬─────┐ │ a ┆ b ┆ c │ │ --- ┆ --- ┆ --- │ │ i64 ┆ list[i64] ┆ u32 │ ╞═════╪═══════════╪═════╡ │ 1 ┆ [1, 2, 3] ┆ 3 │ │ 2 ┆ [2] ┆ 1 │ │ 3 ┆ [5, 0] ┆ 2 │ └─────┴───────────┴─────┘
Question:  The seaborn styles shipped by Matplotlib are deprecated since 3.6, as they no longer correspond to the styles shipped by seaborn. However, they will remain available as 'seaborn-v0_8-<style>'. Alternatively, directly use the seaborn API instead.  I have tried this: # use seaborn style plt.style.use("seaborn")  but it is deprecated, and I want to remove this warning when I use the cmd in windows

Answer: This warning is telling you that seaborn styles in matplotlib do not match current seaborn styles, since the latest have been updated. This is why you should set the style as follow: plt.style.use("seaborn-v0_8")  You can specify a theme by replacing <style> with one the following:  bright colorblind dark dark-palette darkgrid deep muted notebook paper pastel poster talk ticks white whitegrid  Just like this: plt.style.use("seaborn-v0_8-whitegrid")  Alternatively, if you want to use the latest seaborn styles, use their library directly. Edit: adding missing themes for completeness.