eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I'm facing a python timezones problem and am unsure of what is the right approach to deal with it. I have to calculate timedeltas from given start and end DateTime objects. It can happen that daylight saving time will change during the runtime of my events, so I have to take that into account. So far I've learned that for this to work I need to save my start and end times as timezone aware DateTime objects rather than regular UTC DateTimes. I've been looking into DateTime.tzinfo, pytz,and dateutil but from what I understand these are all mostly focused on localised display of UTC DateTime objects or calculating the offsets between different timezones. Other helpers I found expect the timezone as a UTC offset, so would already require me to know if a date is affected by daylight saving or not. So, I guess my question is: Is there a way so save a DateTime as "Central Europe" and have it be aware of daytime savings when doing calculations with them? Or, if not, what would be the established way to check if two DateTime objects are within daylight saving, so I can manually adjust the result if necessary? I'd be grateful for any pointers.

Answer: You just need to produce an aware (localised) datetime instance, then any calculation you do with it will take DST into account. Here as an example with pytz: >>> import pytz >>> from datetime import * >>> berlin = pytz.timezone('Europe/Berlin') >>> d1 = berlin.localize(datetime(2023, 3, 25, 12)) datetime.datetime(2023, 3, 25, 12, 0, tzinfo=<DstTzInfo 'Europe/Berlin' CET+1:00:00 STD>) >>> d2 = berlin.localize(datetime(2023, 3, 26, 12)) datetime.datetime(2023, 3, 26, 12, 0, tzinfo=<DstTzInfo 'Europe/Berlin' CEST+2:00:00 DST>) >>> d2 - d1 datetime.timedelta(seconds=82800) >>> (d2 - d1).total_seconds() / 60 / 60 23.0
Question: I am using following code to create the function and trigger to update the created_at and updated_at fields. with upgrade of new module getting the deprecated API warning. How can I replace  engine.execute(sa.text(create_refresh_updated_at_func.format(schema=my_schema))) line to remove the warning message? Code: mapper_registry.metadata.create_all(engine, checkfirst=True) create_refresh_updated_at_func = """ CREATE OR REPLACE FUNCTION {schema}.refresh_updated_at() RETURNS TRIGGER AS $$ BEGIN NEW.updated_at = NOW(); RETURN NEW; END; $$ LANGUAGE plpgsql; """ my_schema = "public" engine.execute(sa.text(create_refresh_updated_at_func.format(schema=my_schema)))  Warrning:  RemovedIn20Warning: Deprecated API features detected! These feature(s) are not compatible with SQLAlchemy 2.0. To prevent incompatible upgrades prior to updating applications, ensure requirements files are pinned to "sqlalchemy<2.0". Set environment variable SQLALCHEMY_WARN_20=1 to show all deprecation warnings. Set environment variable SQLALCHEMY_SILENCE_UBER_WARNING=1 to silence this message. (Background on SQLAlchemy 2.0 at: https://sqlalche.me/e/b8d9) engine.execute(sa.text(create_refresh_updated_at_func.format(schema=my_schema))) 

Answer: SQLAlchemy no longer supports autocommit at the library level. You need to run the execute within a transaction. This should work: with engine.begin() as conn: conn.execute(text(create_refresh_updated_at_func.format(schema=my_schema)))  migration-core-connection-transaction You could also use the driver-level isolation level like this but I think the connections from this pool will all be set to autocommit: engine2 = create_engine(f"postgresql+psycopg2://{username}:{password}@/{db}", isolation_level='AUTOCOMMIT') with engine2.connect() as conn: conn.execute(text(create_refresh_updated_at_func.format(schema=my_schema)))
Question: Since Python 3.8 it is possible to use self-documenting expressions in f-strings like this: >>> variable=5 >>> print(f'{variable=}') variable=5  is there an equivalent feature in C#?

Answer: Yes. int variable = 5; Console.WriteLine($"variable={variable}");  That outputs: variable=5  The key here is the $ that precedes the string literal.  To do what you want with the name coming dynamically, I'd suggest a more explicit approach of using an extension method. Try this: public static class SelfDocExt { public static string SelfDoc<T>( this T value, [CallerArgumentExpression(nameof(value))] string name = "") => $"{name}={value}"; }  Then you can write this: int variable = 5; Console.WriteLine($"{variable.SelfDoc()}");  It solves your problem without breaking string interpolation.
Question: I am using AWS ECS with a Python Framework and in my task definition i have the option to add environment variables that will be available to the service(cluster). Here is where i added the env variables:  When i then try to print all the env variables in my service i do not get access to these variables and i am not sure why. Here i printed all my env using environ: for a in os.environ: print('Var: ', a, 'Value: ', os.getenv(a)) print("all done")  Result:  DB_PORT or APP_KEY is not available in my service or python-code.

Answer: I had a similar problem and it seems to me that the full environment is passed only to the PID 1 (init process, which in a container should be CMD/ENTRYPOINT command). Cron is not that process so you cannot assume it sees the same environment. What I did may not be the best solution, it is rather a hack, but it works. The environment of a process is available in /proc/<pid>/environ, so in this case /proc/1/environ. I grab it from there and I store it in a file for a future use: for I in `cat /proc/1/environ | strings`; do echo "export $I"; done > /src/.profile  and then I just source /src/.profile in my scripts (the cron job in your case). If you need AWS credentials, you may also need access to ECS_CONTAINER_METADATA_URI_V4 environment variable and that one will be also there.
Question: I am using conda version 4.14.0. When I activate a conda environment I can see that the current numpy module is 1.22.3 conda list | grep -i numpy numpy 1.22.3 py39hc58783e_2 conda-forge  When I run python in the conda environment and load numpy it shows version 1.19.1 Python 3.9.12 | packaged by conda-forge | (main, Mar 24 2022, 23:25:59) [GCC 10.3.0] on linux Type "help", "copyright", "credits" or "license" for more information. >>> import numpy as np >>> np.__version__ '1.19.5'  Looking at the sys.path locations I see that it is searching my local account site packages before the environment site-packages, how can this order be updated? >>> print(sys.path) ['', '/opt/tljh/user/lib/python39.zip', '/opt/main/user/lib/python3.9', '/opt/main/user/lib/python3.9/lib-dynload', '/home/user/.local/lib/python3.9/site-packages', '/opt/main/user/lib/python3.9/site-packages']  Thank you for your help. I cannot find where these variables are added through conda.

Answer: The user site packages (at ~/.local) remaining available is the default behaviour of conda envs, and even issues requesting adding an option to disable that have been repeatedly closed by conda maintainers (example). It is an unfortunate decision from Conda that the user site is included by default. You can exclude it from sys.path by setting an environment variable: export PYTHONNOUSERSITE=1  That sounds hacky but it is actually the recommended workaround from conda themselves (ref). Or, you can do what I do and just keep the user site totally empty, to avoid such issues without needing to mess around with env vars. Uninstall everything from ~/.local and use separate venvs for everything. ~/.local/bin/python3.9 -m pip uninstall -r <(~/.local/bin/python3.9 -m pip freeze)  You can use a project like pipx as a replacement to install any utilities which should be globally available, this avoids polluting the user site (and conda envs) with their dependencies. Watch #7707 (make python user site-packages isolation in .local directory configurable) for possible future developments from conda here.
Question: I have a YML file for a Conda environment that runs with Python 3.8.15 (environment.yml). I am currently trying to load that file into my Google Colab, based on this answer: conda environment in google colab [google-colaboratory]. !wget -c https://repo.anaconda.com/archive/Anaconda3-2022.10-Windows-x86_64.exe !chmod +x Anaconda3-2022.10-Windows-x86_64.exe !bash ./Anaconda3-2022.10-Windows-x86_64.exe -b -f -p /usr/local  And while the executable file for Anaconda was installed in my Google Drive folder, when I run the code, it turms out that Colab could not execute that file: ./Anaconda3-2022.10-Windows-x86_64.exe: ./Anaconda3-2022.10-Windows-x86_64.exe: cannot execute binary file  Is there any other method that I could use to install Anaconda to work with it in Google Colab? And furthermore, how should I load my environment.yml file after getting Anaconda to run in Colab?

Answer: You cannot run a Jupyter notebook (Colab session) with a new Conda environment, but you can use Conda to augment the packages in the existing Python installation. Installation is streamlined with condacolab. See the condacolab documentation. An quick example would go something like: First Cell !pip install -q condacolab import condacolab condacolab.install() # expect a kernel restart  Second Cell mamba install [pkg1 pkg2 ...] # or, if you have a YAML mamba env update -n base -f env.yaml  In the YAML case, you must match the same version of Python with the Colab version.
Question: I'm trying to simplify some expressions of positive odd integers with sympy. But sympy refuses to expand floor, making the simplification hard to proceed. To be specific, x is a positive odd integer (actually in my particular use case, the constraint is even stricter. But sympy can only do odd and positive, which is fine). x // 2 should be always equal to (x - 1) / 2. Example code here: from sympy import Symbol, simplify x = Symbol('x', odd=True, positive=True) expr = x // 2 - (x - 1) / 2 print(simplify(expr))  prints -x/2 + floor(x/2) + 1/2. Ideally it should print 0. What I've tried so far:  Simplify (x - 1) // 2 - (x - 1) / 2. Turns out to be 0. Multiply the whole thing by 2: 2 * (x // 2 - (x - 1) / 2). Gives me: -x + 2*floor(x/2) + 1. Try to put more weights on the FLOOR op by customizing the measure. No luck. Use sympy.core.evaluate(False) context when creating the expression. Nuh. Tune other parameters like ratio, rational, and play with other function like expand, factor, collect. Doesn't work either.  EDIT: Wolfram alpha can do this. I tried to look like the assumptions of x along with some expressions. It surprises me that (x - 1) / 2).is_integer returns None, which means unknown. I'm running out of clues. I'm even looking for alternativese of sympy. Any ideas guys?

Answer: I fail to see why sympy can't simplify that. But, on another hand, I've discovered the existence of odd parameter just now, with your question. What I would have done, without the knowledge of odd is k = Symbol('k', positive=True, integer=True) x = 2*k-1 expr = x // 2 - (x - 1) / 2  Then, expr is 0, without even the need to simplify. So, can't say why you way doesn't work (and why that odd parameter exists if it is not used correctly to guess that x-1 is even, and therefore (x-1)/2 integer). But, in the meantime, my way of defining an odd integer x works.
Question: I am trying to read a large database table with polars. Unfortunately, the data is too large to fit into memory and the code below eventually fails. Is there a way in polars how to define a chunksize, and also write these chunks to parquet, or use the lazy dataframe interface to keep the memory footprint low? import polars as pl df = pl.read_sql("SELECT * from TABLENAME", connection_string) df.write_parquet("output.parquet") 

Answer: Yes and no. There's not a predefined method to do it but you can certainly do it yourself. You'd do something like: rows_at_a_time=1000 curindx=0 while True: df = pl.read_sql(f"SELECT * from TABLENAME limit {curindx},{rows_at_a_time}", connection_string) if df.shape[0]==0: break df.write_parquet(f"output{curindx}.parquet") curindx+=rows_at_a_time ldf=pl.concat([pl.scan_parquet(x) for x in os.listdir(".") if "output" in x and "parquet" in x])  This borrows limit syntax from this answer assuming you're using mysql or a db that has the same syntax which isn't trivial assumption. You may need to do something like this if not using mysql. Otherwise you just read your table in chunks, saving each chunk to a local file. When the chunk you get back from your query has 0 rows then it stops looping and loads all the files to a lazy df. You can almost certainly (and should) increase the rows_at_a_time to something greater than 1000 but that's dependent on your data and computer memory.
Question: I have the following dataframe print(A) Index 1or0 0 1 0 1 2 0 2 3 0 3 4 1 4 5 1 5 6 1 6 7 1 7 8 0 8 9 1 9 10 1  And I have the following Code (Pandas Dataframe count occurrences that only happen immediately), which counts the occurrences of values that happen immediately one after another. ser = A["1or0"].ne(A["1or0"].shift().bfill()).cumsum() B = ( A.groupby(ser, as_index=False) .agg({"Index": ["first", "last", "count"], "1or0": "unique"}) .set_axis(["StartNum", "EndNum", "Size", "Value"], axis=1) .assign(Value= lambda d: d["Value"].astype(str).str.strip("[]")) ) print(B) ​ StartNum EndNum Size Value 0 1 3 3 0 1 4 7 4 1 2 8 8 1 0 3 9 10 2 1  The issues is, when NaN Values occur, the code doesn't put them together in one interval it count them always as one sized interval and not e.g. 3 print(A2) Index 1or0 0 1 0 1 2 0 2 3 0 3 4 1 4 5 1 5 6 1 6 7 1 7 8 0 8 9 1 9 10 1 10 11 NaN 11 12 NaN 12 13 NaN   print(B2) ​ StartNum EndNum Size Value 0 1 3 3 0 1 4 7 4 1 2 8 8 1 0 3 9 10 2 1 4 11 11 1 NaN 5 12 12 1 NaN 6 13 13 1 NaN  But I want B2 to be the following print(B2Wanted) ​ StartNum EndNum Size Value 0 1 3 3 0 1 4 7 4 1 2 8 8 1 0 3 9 10 2 1 4 11 13 3 NaN  What do I need to change so that it works also with NaN?

Answer: First fillna with a value this is not possible (here -1) before creating your grouper: group = A['1or0'].fillna(-1).diff().ne(0).cumsum() # or # s = A['1or0'].fillna(-1) # group = s.ne(s.shift()).cumsum() B = (A.groupby(group, as_index=False) .agg(**{'StartNum': ('Index', 'first'), 'EndNum': ('Index', 'last'), 'Size': ('1or0', 'size'), 'Value': ('1or0', 'first') }) )  Output:  StartNum EndNum Size Value 0 1 3 3 0.0 1 4 7 4 1.0 2 8 8 1 0.0 3 9 10 2 1.0 4 11 13 3 NaN
Question: I have a data set with three columns. Column A is to be checked for strings. If the string matches foo or spam, the values in the same row for the other two columns L and G should be changed to XX. For this I have tried the following. df = pl.DataFrame( { "A": ["foo", "ham", "spam", "egg",], "L": ["A54", "A12", "B84", "C12"], "G": ["X34", "C84", "G96", "L6",], } ) print(df) shape: (4, 3) ┌──────┬─────┬─────┐ │ A ┆ L ┆ G │ │ --- ┆ --- ┆ --- │ │ str ┆ str ┆ str │ ╞══════╪═════╪═════╡ │ foo ┆ A54 ┆ X34 │ │ ham ┆ A12 ┆ C84 │ │ spam ┆ B84 ┆ G96 │ │ egg ┆ C12 ┆ L6 │ └──────┴─────┴─────┘  expected outcome shape: (4, 3) ┌──────┬─────┬─────┐ │ A ┆ L ┆ G │ │ --- ┆ --- ┆ --- │ │ str ┆ str ┆ str │ ╞══════╪═════╪═════╡ │ foo ┆ XX ┆ XX │ │ ham ┆ A12 ┆ C84 │ │ spam ┆ XX ┆ XX │ │ egg ┆ C12 ┆ L6 │ └──────┴─────┴─────┘  I tried this df = df.with_columns( pl.when((pl.col("A") == "foo") | (pl.col("A") == "spam")) .then((pl.col("L")= "XX") & (pl.col( "G")= "XX")) .otherwise((pl.col("L"))&(pl.col( "G"))) )  However, this does not work. Can someone help me with this?

Answer: For setting multiple columns to the same value you could use: df.with_columns( pl.when(pl.col("A").is_in(["foo", "spam"])) .then(pl.lit("XX")) .otherwise(pl.col("L", "G")) .name.keep() )  shape: (4, 3) ┌──────┬─────┬─────┐ │ A ┆ L ┆ G │ │ --- ┆ --- ┆ --- │ │ str ┆ str ┆ str │ ╞══════╪═════╪═════╡ │ foo ┆ XX ┆ XX │ │ ham ┆ A12 ┆ C84 │ │ spam ┆ XX ┆ XX │ │ egg ┆ C12 ┆ L6 │ └──────┴─────┴─────┘   .is_in() can be used instead of multiple == x | == y chains  If you need different values, you can pack them into a struct and extract/unnest the fields. df.with_columns( pl.when(pl.col("A").is_in(["foo", "spam"])) .then(pl.struct(L = pl.lit("AAA"), G = pl.lit("BBB"))) .otherwise(pl.struct("L", "G")) .struct.field("L", "G") # or .struct.unnest() )  shape: (4, 3) ┌──────┬─────┬─────┐ │ A ┆ L ┆ G │ │ --- ┆ --- ┆ --- │ │ str ┆ str ┆ str │ ╞══════╪═════╪═════╡ │ foo ┆ AAA ┆ BBB │ │ ham ┆ A12 ┆ C84 │ │ spam ┆ AAA ┆ BBB │ │ egg ┆ C12 ┆ L6 │ └──────┴─────┴─────┘
Question: Apart from typing out commands - is there a good way to install requirements.txt inside VSCode. I have a workspace with 2 folders containing different Python projects added - each has it's own virtual environment. I would like to run a task to execute and install the requirements to each of these. I have tried adding a task to tasks.json to try and do it for one with no success. { "version": "2.0.0", "tasks": [ { "label": "Service1: Install requirements", "type": "shell", "runOptions": {}, "command": "'${workspaceFolder}/sources/api/.venv/Scripts/activate'; pip install -r '${workspaceFolder}/sources/api/requirements.txt'", "problemMatcher": [] } ] }  This runs - but you can see it refer to my global Python packages h:\program files\python311\lib\site-packages - not the virtual environment. I am running on Windows for this - but would like it to work eventually with Linux.

Answer: I've written a more detailed post before, but as Andez mentioned in comments, this is also a suitable post for the answer. This task can be ran in Windows, Linux and MacOS. { "version": "2.0.0", "tasks": [ { "label": "Build Python Env", "type": "shell", "group": { "kind": "build", "isDefault": true }, "options": { "cwd": "${workspaceFolder}" }, "linux": { "command": "python3 -m venv py_venv && source py_venv/bin/activate && python3 -m pip install --upgrade pip && python3 -m pip install -r requirements.txt && deactivate py_venv" }, "osx": { "command": "python3 -m venv py_venv && source py_venv/bin/activate && python3 -m pip install --upgrade pip && python3 -m pip install -r requirements.txt && deactivate py_venv" }, "windows": { "options": { "shell": { "executable": "C:\\Windows\\system32\\cmd.exe", "args": [ "/d", "/c" ] }, }, "command": "(if not exist py_venv py -m venv py_venv) && .\\py_venv\\Scripts\\activate.bat && py -m pip install --upgrade pip && py -m pip install -r requirements.txt && deactivate py_venv" }, "problemMatcher": [] } ] }
Question: Is joblib (https://joblib.readthedocs.io/en/latest/index.html) expected to be reliable across different machines, or ways of running functions, even different sessions on the same machine over time? For concreteness if you run this code in a Jupyter notebook, or as a python script, or piped to the stdin a python interpreter, you get different cache entries. The piped version seems to be a special case where you get a JobLibCollisionWarning that leads to it running every time and never reading from the cache. The other two though, end up having a different path saved in the joblib cache dir, and inside each one the same hash directory (fb65b1dace3932d1e66549411e3310b6) exists. from joblib import Memory memory = Memory('./cache', verbose=0) @memory.cache def job(x): print(f'Running with {x}') return x**2 print(job(2))  you get multiple cache entries. These entries also are in folders that contain path information (including what appears to be a tmp directory for the notebook entry, e.g. main--var-folders-3q-ht_2mtk52hl7ydxrcr87z2gr0000gn-T-ipykernel-3189892766), so it looks like if I transferred to another machine that the jobs would all be run again. I don't know how that path is reliable in the long run, it seems likely the tmpdir could change, or the ipykernel could have some other number associated with it. Is this expected?

Answer: The canonical way of using joblib's disk caching seems to require always having your function in a .py file, and not in a notebook cell (see e.g. this issue). I've found a workaround for using joblib in a jupyter notebook anyway, so that it hits the cache even if you re-run a cell or restart the notebook (which does not happen by default, as you've found). It is to manually set the __module__ of your function to some unique identifier (e.g, the notebook name). The following is a wrapper for Memory.cache that does that: def cache(mem, module, **mem_kwargs): def cache_(f): f.__module__ = module f.__qualname__ = f.__name__ return mem.cache(f, **mem_kwargs) return cache_  Usage, with your example function: from joblib import Memory mem = Memory('./cache') @cache(mem, "my_notebook_name") def job(x): print(f'Running with {x}') return x**2  This will save the function's outputs in ./cache/my_notebook_name/job/. (Had this idea while reading joblib's source, specifically get_func_name in func_inspect.py, in which the function's __module__ and __name__/__qualname__ are read).
Question: I was building a documentation site for my python project using mkdocstrings. For generating the code referece files I followed this instructions https://mkdocstrings.github.io/recipes/ I get these errors:  INFO - Building documentation... INFO - Cleaning site directory INFO - The following pages exist in the docs directory, but are not included in the "nav" configuration: - reference\SUMMARY.md - reference_init_.md ... ... - reference\tests\manual_tests.md ERROR - mkdocstrings: No module named ' ' ERROR - Error reading page 'reference/init.md': ERROR - Could not collect ' '  This is my file structure:  This is my docs folder:  I have the same gen_ref_pages.py file shown in the page:  from pathlib import Path import mkdocs_gen_files nav = mkdocs_gen_files.Nav() for path in sorted(Path("src").rglob("*.py")): module_path = path.relative_to("src").with_suffix("") doc_path = path.relative_to("src").with_suffix(".md") full_doc_path = Path("reference", doc_path) parts = tuple(module_path.parts) if parts[-1] == "__init__": parts = parts[:-1] elif parts[-1] == "__main__": continue nav[parts] = doc_path.as_posix() # with mkdocs_gen_files.open(full_doc_path, "w") as fd: ident = ".".join(parts) fd.write(f"::: {ident}") mkdocs_gen_files.set_edit_path(full_doc_path, path) with mkdocs_gen_files.open("reference/SUMMARY.md", "w") as nav_file: # nav_file.writelines(nav.build_literate_nav()) # ``` This is my mkdocs.yml: ``` site_name: CA Prediction Docs theme: name: "material" palette: primary: deep purple logo: assets/logo.png favicon: assets/favicon.png features: - navigation.instant - navigation.tabs - navigation.expand - navigation.top # - navigation.sections - search.highlight - navigation.footer icon: repo: fontawesome/brands/git-alt copyright: Copyright &copy; 2022 - 2023 Ezequiel González extra: social: - icon: fontawesome/brands/github link: https://github.com/ezegonmac - icon: fontawesome/brands/linkedin link: https://www.linkedin.com/in/ezequiel-gonzalez-macho-329583223/ repo_url: https://github.com/ezegonmac/TFG-CellularAutomata repo_name: ezegonmac/TFG-CellularAutomata plugins: - search - gen-files: scripts: - docs/gen_ref_pages.py - mkdocstrings nav: - Introduction: index.md - Getting Started: getting-started.md - API Reference: reference.md # - Reference: reference/ - Explanation: explanation.md 

Answer: Following the mkdocstrings documentation I also received that same error. After some tinkering, I was able to successfully serve the docs by adding paths: [src] (see below). mkdocs.yml plugins: - search - gen-files: scripts: - docs/gen_ref_pages.py - mkdocstrings: default_handler: python handlers: python: paths: [src]