eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: Wish to re-arrange date columns (with Month Year as header) in descending order from left to right. All non-date cols are shifted to extreme left before the date columns begin. If possible as the Column headers are mix-case, so need to process as case-insensitive. demo = { 'Name': ['Alice', 'Bob', 'Charlie', 'David'], 'Age': [25, 30, 35, 40], 'Apr 2022': ['New', 'Los', 'Chic', 'Hous'], 'aPRil 2024': ['York', 'Anges', 'cago', 'ston'], 'Vanilla': ['nfd', 'bdfh', 'tyii', 'liu'], 'aUg 2023': ['NewYork', 'LosAngeles', 'Chicago', 'Houston'], 'deC 2022': ['Neork', 'Logeles', 'Chago', 'Hoston'] } result = { 'Name': ['Alice', 'Bob', 'Charlie', 'David'], 'Age': [25, 30, 35, 40], 'Vanilla': ['nfd', 'bdfh', 'tyii', 'liu'], 'aPRil 2024': ['York', 'Anges', 'cago', 'ston'], 'aUg 2023': ['NewYork', 'LosAngeles', 'Chicago', 'Houston'], 'deC 2022': ['Neork', 'Logeles', 'Chago', 'Hoston'], 'Apr 2022': ['New', 'Los', 'Chic', 'Hous'] } 

Answer: You can convert to_datetime with errors='coerce' as key to sort_index. In addition pass ascending=False, na_position='first' as parameters to get the dates in descending order and the non-dates first: df = pd.DataFrame(demo) result = df.sort_index( axis=1, key=lambda x: pd.to_datetime(x, errors='coerce', format='mixed'), ascending=False, na_position='first', )  Note depending on the pandas version, it might be needed to first preprocess the string to homogenize the case: key=lambda x: pd.to_datetime(x.str.title(), errors='coerce'). Output:  Name Age Vanilla aPRil 2024 aUg 2023 deC 2022 Apr 2022 0 Alice 25 nfd York NewYork Neork New 1 Bob 30 bdfh Anges LosAngeles Logeles Los 2 Charlie 35 tyii cago Chicago Chago Chic 3 David 40 liu ston Houston Hoston Hous  This will use the following intermediate to sort the columns: pd.to_datetime(df.columns, errors='coerce', format='mixed') DatetimeIndex(['NaT', 'NaT', '2022-04-01', '2024-04-01', 'NaT', '2023-08-01', '2022-12-01'], dtype='datetime64[ns]', freq=None)
Question: I made a form that inherits from the UserCreationForm and use class based view that inherits CreateView and when I use runserver and display the form, there is a section at the bottom Password-based authentication that I don't notice forms.py from django.contrib.auth import get_user_model from django.contrib.auth.forms import UserCreationForm class RegisterForm(UserCreationForm): """Form to Create new User""" class Meta: model = get_user_model() fields = ["username", "password1", "password2"]  views.py from django.views.generic import CreateView from .forms import RegisterForm from django.urls import reverse_lazy class SignUp(CreateView): form_class = RegisterForm template_name = "register.html" success_url = reverse_lazy("core:Login") def form_valid(self, form): user = form.save() if user: login(self.request, user) return super().form_valid(form)  register.html <h1>signup</h1> {{form}}  And when I ran the code I saw this output  So I didn't expect password-based authentication. My question is about  What exactly is this? Should it be displayed here? How do I hide it? 

Answer: From Django version 5.1 onwards the UserCreationForm has a usable_password field by default. This relates to the feature Django has for setting unusable passwords for users. This is useful in case you're using some kind of external authentication like Single Sign-On or LDAP. Since your form seems to be a user facing one and showing this field to your user doesn't make much sense (this particular form is by default geared more towards usage in the admin site) you should simply remove the field from your form by setting it to None: class RegisterForm(UserCreationForm): """Form to Create new User""" usable_password = None class Meta: model = get_user_model() fields = ["username", "password1", "password2"]
Question: My DataFrame is: import pandas as pd df = pd.DataFrame( { 'a': [20, 30, 2, 5, 10] } )  Expected output is pct_change() of a:  a pct_change 0 20 -50.000000 1 30 50.000000 2 2 -93.333333 3 5 150.000000 4 10 100.000000  I want to compare df.a.iloc[0] with 40 for the first value of pct_change. If I use df['pct_change'] = df.a.pct_change().mul(100), the first value is NaN. My Attempt: def percent(a, b): result = ((a - b) / b) * 100 return result.round(2) df.loc[df.index[0], 'pct_change'] = percent(df.a.iloc[0], 40)  Is there a better/more efficient way?

Answer: You can use the fill_value keyword argument in pct_change. The pct_change documentation says:  Additional keyword arguments are passed into DataFrame.shift or Series.shift.  and Series.shift accepts a fill_value argument to fill missing rows. import pandas as pd df = pd.DataFrame({"a": [20, 30, 2, 5, 10]}) df["pct_change"] = df["a"].pct_change(fill_value=40).mul(100) print(df)  Output:  a pct_change 0 20 -50.000000 1 30 50.000000 2 2 -93.333333 3 5 150.000000 4 10 100.000000
Question: I have a python virtual environment in /Documents and a project in /Documents/Code/Python/example. I want to use this venv in my project but I can't get the vs code python interpreter to recognize that the venv in /Documents is a venv. I tried to use set the interpreter path by using the "Find..." button to manually select /Documents/.venv/bin/python with the finder. However, it is recognizing it as something other than a virtual environment as it looks different when compared to a venv in the same directory as the project. Here is the output below: 2024-08-08 16:53:55.325 [info] Discover tests for workspace name: example - uri: /Users/varun/Documents/Code/Python/example 2024-08-08 16:53:55.326 [info] Python interpreter path: /opt/homebrew/Cellar/python@3.12/3.12.4/Frameworks/Python.framework/Versions/3.12/bin/python3.12  How it looks when venv is in a parent directory How it looks when venv is in same directory I suspect that it is because /Documents/.venv/bin/python is referencing a "faulty" version of python (I'm not sure though). I'm able to activate and use the venv from the terminal to run the code but with vs code not recognizing the venv it isn't giving me any autofill suggestions. Please let me know if you need more info, thanks.

Answer: it could be that vscode didn't find the correct interpreter path. Are you creating a virtual environment using the command python -m venv myvenv? Or try restarting vscode after clearing the cache and then opening it up to create a new virtual environment. If you still can't recognize it, you can set the path to the virtual environment in the .vscode/settings.json file. Add "python.pythonPath":"path/to/your/project/venv/bin/python" and  "python.defaultInterpreterPath": "path/to/your/venv/bin/python". See if that solves the problem.
Question: I wish to use shelve in an asyncio program and I fear that every change will cause the main event loop to stall. While I don't mind the occasional slowdown of the pickling operation, the disk writes may be substantial. Every how often does shelve sync to disk? Is it a blocking operation? Do I have to call .sync()? If I schedule the sync() to run under a different thread, a different asyncio task may modify the shelve at the same time, which violates the requirement of single-thread writes.

Answer: shelve, by default, is backed by the dbm module, in turn backed by some dbm implementation available on the system. Neither the shelve module, nor the dbm module, make any effort to minimize writes; an assignment of a value to a key causes a write every time. Even when writeback=True, that just means that new assignments are placed in the cache and immediately written to the backing dbm; they're written to make sure the original value is there, and the cache entry is made because the object assigned might change after assignment and needs to be handled just like a freshly read object (meaning it will be written again when synced or closed, in case it changed). While it's possible some implementation of the underlying dbm libraries might include some caching, AFAICT, most do try to write immediately (that is, pushing data to the kernel immediately without user-mode buffering), they just don't necessarily force immediate synchronization to disk (though it can be requested, e.g. with gdbm_sync). writeback=True will make it worse, because when it does sync, it's a major effort (it literally rewrites every object read or written to the DB since the last sync, because it has no way of knowing which of them might have been modified), as opposed to the small effort of rewriting a single key/value pair at a time. In short, if you're really concerned about blocking writes, you can't use unthreaded async code without potential blocking, but said blocking is likely short-lived as long as writeback=True is not involved (or as long as you don't sync/close it until performance considerations are no longer relevant). If you need to have truly non-blocking async behavior, all shelve interactions will need to occur under a lock in worker threads, and either writeback must be False (to avoid race conditions pickling data) or if writeback is True, you must take care to avoid modifying any object that might be in the cache during the sync/close.
Question: There is a way to generate a HTML high level graph in a jupyter notebook as shown in dasks' documentation: https://docs.dask.org/en/stable/graphviz.html#high-level-graph-html-representation Taking the example from the docs, you put the following code in a jupyter cell import dask.array as da x = da.ones((15, 15), chunks=(5, 5)) y = x + x.T y.dask # shows the HTML representation in a Jupyter notebook  And you get a nice interactive html view of the graph in the jupyter notebook. My question is if there a way to get the html from this graph outside of the jupyter context. My immediate interest is to export a static html file to disk as a record of the graph that was executed for a task. I could also see other applications, such as embedding a widget in a gui.

Answer: By default, the Notebook interface will display the _repr_html_() method's output for whatever it's trying to display. In the case of a Dask Array, the dask attribute is an instance of a HighLevelGraph, whose implementation is here: https://github.com/dask/dask/blob/ed5f68897b3a097f7c5ec1a9ec13ce49c112a544/dask/highlevelgraph.py#L840 That method should return a string, so you can call it directly and save the output to a file: from pathlib import Path Path("dask.html").write_text(y.dask._repr_html_())  There are also ways to use the IPython APIs to run through the process that the Notebook kernel actually goes through when it goes to display some data, but I didn't look those up 😀
Question: I'm writing an Excel using XLSXWRITER package in Python. Formats are applied to cells and this all works, except for one cell when the value assigned is a 27 digit text string (extracted from some source). I've read How to apply format as 'Text' and 'Accounting' using xlsxwriter. It suggests to set the number format to '@', but when I try: WSFMT_TX_REFINFO = wb.add_format({'num_format': '@' , 'align': 'right' , 'border_color': WS_TX_BORDERCOLOR , 'right': WS_TX_BORDERSYTLE })  and write a cell with: refdata = '001022002024080400002400105' ws.write(wsRow, WS_COL_REF_INFO, refdata, WSFMT_TX_REFINFO)  The cell is shown as 1.041E+24  and in the editor field as 1.041002024073E+24  If I change the format specification from '@' to 0, i.e. change WSFMT_TX_REFINFO = wb.add_format({'num_format': '@'  to WSFMT_TX_REFINFO = wb.add_format({'num_format': 0  the cell is shown as 1022002024080400000000000  Note that the digits after the 14th are replaced by zeros. In the editor field it shows as 1.0220020240804E+24  What I need: The number shall be show as 27 digit string, exactly as found in refdata Note: There are cases, where refdata may contain alphanumeric strings in some cases, besides pure 27 digit strings. Any hint?

Answer: This may be a case where it is better to use write_string() instead of the more generic write().
Question: I have several local time points: import datetime from zoneinfo import ZoneInfo as zi wmr = datetime.time(hour=12, tzinfo=zi("GMT")) ecb = datetime.time(hour=14, minute=15, tzinfo=zi("CET")) jpx = datetime.time(hour=14, tzinfo=zi("Japan"))  which I want to convert to UTC times given a date. E.g., local2utc(datetime.datetime(2024,1,1), wmr) ---> "2024-01-01 12:00:00" local2utc(datetime.datetime(2024,6,1), wmr) ---> "2024-06-01 11:00:00" (DST active) local2utc(datetime.datetime(2024,1,1), ecb) ---> "2024-01-01 13:15:00" local2utc(datetime.datetime(2024,6,1), ecb) ---> "2024-06-01 12:15:00" (DST active) local2utc(datetime.datetime(2024,1,1), jpx) ---> "2024-01-01 05:00:00" local2utc(datetime.datetime(2024,6,1), jpx) ---> "2024-06-01 05:00:00" (no DST in Japan)  The following implementation def local2utc(date, time): local_dt = datetime.datetime.combine(date,time) tm = local_dt.utctimetuple() return datetime.datetime(*tm[:7])  seems for work for Japan and CET, but not for GMT/WET (because London is on BST/WEST in the summer). So, what do I do?

Answer: Python uses the IANA time zone database. The list of time zone names can be found here. According to this table, "GMT" is a time zone that has a 0 UTC offset and does not observe daylight saving. Perhaps "Europe/London" would give you the results you are looking for.
Question: I have the following regex.replace: self.reply = raw_reply.replace(b"<" + rcid + b":", b"")  where rcid is a command reference. raw_reply is a huge mass of data in bytes e.g. <35:\x07\x98c\x45\x09 etc.  I want it to remove all instances of, for example <35: but only if one has not been replaced less than 1024 characters ago. Is there a way to do this with regex? I've tried looking at exclusions and negative lookahead but not sure how to implement it when i want it to ignore any matches within 1024 characters of the previous match.

Answer: Use a regular expression that matches up to 1024 characters after the pattern you're replacing. Capture the excess 1024 characters in a capture group so you can copy them to the replacement. The next match will have to be after this, since overlapping matches are not processed. self.reply = re.sub(b"<" + rcid + b":(.{,1024})", br"\1", raw_reply, flags=re.DOTALL)
Question: I need to convert a list (including headers) to a Dataframe. If I do it directly using pl.DataFrame(list), the headers are created and everything is kept as a string. Moreover, the table is transposed, such that the first element in the list becomes the first column in the dataframe. Input list. [ ['Earnings estimate', 'Current qtr. (Jun 2024)', 'Next qtr. (Sep 2024)', 'Current year (2024)', 'Next year (2025)'], ['No. of analysts', '13', '11', '26', '26'], ['Avg. Estimate', '1.52', '1.62', '6.27', '7.23'], ['Low estimate', '1.36', '1.3', '5.02', '5.88'], ['High estimate', '1.61', '1.74', '6.66', '8.56'], ['Year ago EPS', '1.76', '1.36', '5.74', '6.27'], ]  Expected output. 

Answer: You can explicitly define the orient= to prevent transposition: pl.DataFrame(data[1:], orient="row", schema=data[0])  shape: (5, 5) ┌───────────────────┬─────────────────────────┬──────────────────────┬─────────────────────┬──────────────────┐ │ Earnings estimate ┆ Current qtr. (Jun 2024) ┆ Next qtr. (Sep 2024) ┆ Current year (2024) ┆ Next year (2025) │ │ --- ┆ --- ┆ --- ┆ --- ┆ --- │ │ str ┆ str ┆ str ┆ str ┆ str │ ╞═══════════════════╪═════════════════════════╪══════════════════════╪═════════════════════╪══════════════════╡ │ No. of analysts ┆ 13 ┆ 11 ┆ 26 ┆ 26 │ │ Avg. Estimate ┆ 1.52 ┆ 1.62 ┆ 6.27 ┆ 7.23 │ │ Low estimate ┆ 1.36 ┆ 1.3 ┆ 5.02 ┆ 5.88 │ │ High estimate ┆ 1.61 ┆ 1.74 ┆ 6.66 ┆ 8.56 │ │ Year ago EPS ┆ 1.76 ┆ 1.36 ┆ 5.74 ┆ 6.27 │ └───────────────────┴─────────────────────────┴──────────────────────┴─────────────────────┴──────────────────┘
Question: The polars user guide suggests that enums have a physical, integer representation. Is it possible to access the integers associated with an enum value? For example, is there a nicer way to get the integer representation in the following example? import numpy as np import polars as pl np.random.seed(556) enum_vals = [ "".join([chr(c_code) for c_code in np.random.randint(97, 123, 3)]) for n in range(10) ] enum_dtype = pl.Enum(pl.Series(enum_vals)) ( pl.Series( "enum_vals", [enum_vals[x] for x in np.random.randint(0, len(enum_vals), 5)], dtype=enum_dtype, ) .to_frame() .with_columns( enum_repr=pl.col("enum_vals").map_elements( lambda x: enum_vals.index(x), return_dtype=pl.Int64() ) ) )  shape: (5, 2) ┌───────────┬───────────┐ │ enum_vals ┆ enum_repr │ │ --- ┆ --- │ │ enum ┆ i64 │ ╞═══════════╪═══════════╡ │ loo ┆ 8 │ │ sby ┆ 5 │ │ cqm ┆ 3 │ │ cbn ┆ 2 │ │ vtk ┆ 9 │ └───────────┴───────────┘ 

Answer: For this, pl.Expr.to_physical exists. The documentation also lists the dtype of the underlying physical representation, which is pl.UInt32 for pl.Categorical / pl.Enum. df.with_columns( pl.col("enum_vals").to_physical() )  shape: (5, 1) ┌───────────┐ │ enum_vals │ │ --- │ │ u32 │ ╞═══════════╡ │ 8 │ │ 5 │ │ 3 │ │ 2 │ │ 9 │ └───────────┘
Question: When I type out something like np. I think this triggers Visual Studio Code + Pylance's (not sure) auto-import completion by suggesting that import numpy as np might be relevant. I would like to create similar custom auto-import/complete associations. For example: between pl and polars, so that if I type something like pl. then import polars as pl is given as an auto-import suggestion. How can I do this? Is this specific to the Pylance extension I am using, or something about Visual Studio Code? Please note that auto-import/import-completion is very different from custom code snippets, as covered in How to add custom code snippets in VSCode? The reason for this being:  VS Code adds a new import statement (and therefore has to figure out if it is possible to resolve that import) at the top of the file. It does not add code where the cursor is, which is what a snippet would do. This functionality relies on a language server of some sort (hence my suspicion it is Pylance that is providing this functionality) both to resolve the import, and to insert the import statement at the appropriate location in the file. 

Answer: This is tracked by an open enhancement request: Allow auto-import abbreviations to be configured #2589. I suggest that you give that discussion an upvote to show support for it. You can also subscribe to it to get notified about discussion and progress. Please avoid making noisy comments there like ones that just consist of "+1" / "bump".
Question: I'm not getting the expected value returned from the below code. from playwright.sync_api import sync_playwright import time import random def main(): with sync_playwright() as p: browser = p.firefox.launch(headless=False) page = browser.new_page() url = "https://www.useragentlist.net/" page.goto(url) time.sleep(random.uniform(2,4)) test = page.locator('xpath=//span[1][@class="copy-the-code-wrap copy-the-code-style-button copy-the-code-inside-wrap"]/pre/code/strong').inner_text() print(test) count = page.locator('xpath=//span["copy-the-code-wrap copy-the-code-style-button copy-the-code-inside-wrap"]/pre/code/strong').count() print(count) browser.close() if __name__ == '__main__': main()  page.locator().count() returns a value of 0, I have no issue getting the text from the lines above it, but I need to access all elements, what is wrong with my implementation of locator and count?

Answer: Your second locator XPath has no @class=, so it's different than the first one that works. Store the string in a variable so you don't have to type it twice or encounter copy-paste or stale data errors. In any case, your approach seems overcomplicated. Each user agent is in a <code> tag--just scrape that: from playwright.sync_api import sync_playwright # 1.44.0 def main(): with sync_playwright() as p: browser = p.firefox.launch() page = browser.new_page() url = "https://www.useragentlist.net/" page.goto(url, wait_until="domcontentloaded") agents = page.locator("code").all_text_contents() print(agents) browser.close() if __name__ == "__main__": main()  Locators auto-wait so there's no need to sleep. Avoid XPaths 99% of the time--they're brittle and difficult to read and maintain. Just use CSS selectors or user-visible locators. The goal is to choose the simplest selector necessary to disambiguate the elements you want, and nothing more. span/pre/code/strong is a rigid hierarchy--if one of these changes, your code breaks unnecessarily. By the way, the user agents are in the static HTML, so unless you're trying to circumvent a block, you can do this faster with requests and Beautiful Soup: from requests import get # 2.31.0 from bs4 import BeautifulSoup # 4.10.0 response = get("https://www.useragentlist.net") response.raise_for_status() print([x.text for x in BeautifulSoup(response.text, "lxml").select("code")])  Better still (possibly), use a library like fake_useragent to generate your random user agent.
Question: After using a Pool from Python's multiprocessing to parallelize some computationally intensive work, I wish to retrieve statistics that were kept local to each spawned process. Specifically, I have no real-time interest in these statistics, so I do not want to bear the overhead that would be involved with using a synchronized data structure in which statistics could be kept. I've found some suggestions where the idea would be to use a second pool.map() with a different function which returns the state local to its worker. I believe this to be incorrect since there is no guarantee that this second invocation would lead to exactly one job being distributed to every worker process in the pool. Is there a mechanism that would achieve this? Skeleton snippet where it's unclear what can be done after the imap_unordered() completes. import multiprocessing as mp import random local_stats = {"success": 0, "fails": 0} def do_work(_): if random.choice([True, False]): local_stats["success"] += 1 else: local_stats["fails"] += 1 if __name__ == "__main__": with mp.Manager() as manager: with mp.Pool(processes=2) as pool: results = list(pool.imap_unordered(do_work, range(1000))) # after .imap_unordered() completes, aggregate "local_stats" from each process in the pool # by either retrieving its local_stats, or having them push those stats to the main process # ??? 

Answer: IDK if this is the best solution (could you just log to a file as you go, then parse the files from each child afterwards?), but you mentioned ensuring a task is evenly distributed to all workers. This would commonly be achieved with a Barrier. It is somewhat difficult to pass certain things to child processes like locks queues and such so we will pass it as an argument to the initialization function for all child processes. Here's an example: import multiprocessing as mp import random local_stats = {"success": 0, "fails": 0} def do_work(_): global local_stats if random.choice([True, False]): local_stats["success"] += 1 else: local_stats["fails"] += 1 def init_pool(barrier): # save barrier to child process globals as pool init (barrier must be sent to child at process creation) global sync_barrier sync_barrier = barrier def return_stats(_): # needs dummy arg to call with "map" functions global sync_barrier sync_barrier.wait() # wait for other processes to also be waiting # may raise BrokenBarrierError global local_stats return local_stats if __name__ == "__main__": nprocs = 2 # re-use number of processes with barrier and pool constructor to make sure you use the same number barrier = mp.Barrier(nprocs) with mp.Pool(processes=nprocs, initializer=init_pool, initargs=(barrier,)) as pool: results = list(pool.imap_unordered(do_work, range(1000))) stats = list(pool.imap_unordered(return_stats, range(nprocs), 1)) # force chunksize to 1 to ensure 1 task per child process print(stats)