eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I see it's possible to append using the series namespace (https://stackoverflow.com/a/70599059/5363883). What I'm wondering is if there is a similar method for appending or concatenating DataFrames. In pandas historically it could be done with df1.append(df2). However that method is being deprecated (if it hasn't already been deprecated) for pd.concat([df1, df2]). Sample frames: df1 = pl.from_repr(""" ┌─────┬─────┬─────┐ │ a ┆ b ┆ c │ │ --- ┆ --- ┆ --- │ │ i64 ┆ i64 ┆ i64 │ ╞═════╪═════╪═════╡ │ 1 ┆ 2 ┆ 3 │ └─────┴─────┴─────┘ """) df2 = pl.from_repr(""" ┌─────┬─────┬─────┐ │ a ┆ b ┆ c │ │ --- ┆ --- ┆ --- │ │ i64 ┆ i64 ┆ i64 │ ╞═════╪═════╪═════╡ │ 4 ┆ 5 ┆ 6 │ └─────┴─────┴─────┘ """)  Desired result: shape: (2, 3) ┌─────┬─────┬─────┐ │ a ┆ b ┆ c │ │ --- ┆ --- ┆ --- │ │ i64 ┆ i64 ┆ i64 │ ╞═════╪═════╪═════╡ │ 1 ┆ 2 ┆ 3 │ │ 4 ┆ 5 ┆ 6 │ └─────┴─────┴─────┘ 

Answer: There are different append strategies depending on your needs. df1 = pl.DataFrame({"a": [1], "b": [2], "c": [3]}) df2 = pl.DataFrame({"a": [4], "b": [5], "c": [6]}) # new memory slab new_df = pl.concat([df1, df2], rechunk=True) # append free (no memory copy) new_df = df1.vstack(df2) # try to append in place df1.extend(df2)  To understand the differences, it is important to understand polars memory is immutable iff it has any copy. Copies in polars are free, because it only increments a reference count of the backing memory buffer instead of copying the data itself. However, if a memory buffer has no copies yet, e.g. the refcount == 1, we can mutate polars memory. Knowing this background there are the following ways to append data:  concat -> concatenate all given DataFrames. This is sort of a linked list of DataFrames. If you pass rechunk=True, all memory will be reallocated to contiguous chunks. vstack -> Adds the data from other to DataFrame by incrementing a refcount. This is super cheap. It is recommended to call rechunk after many vstacks. Or simply use pl.concat. extend This operation copies data. It tries to copy data from other to DataFrame. If however the refcount of DataFrame is larger than 1. A new buffer of memory is allocated to hold both DataFrames.
Question: In Python, is there a way to distinguish between strings and other iterables of strings? A str is valid as an Iterable[str] type, but that may not be the correct input for a function. For example, in this trivial example that is intended to operate on sequences of filenames: from typing import Iterable def operate_on_files(file_paths: Iterable[str]) -> None: for path in file_paths: ...  Passing in a single filename would produce the wrong result but would not be caught by type checking. I know that I can check for string or byte types at runtime, but I want to know if it's possible to catch silly mistakes like that with a type-checking tool. I've looked over the collections.abc module and there doesn't seem to be any abc that would include typical iterables (e.g. lists, tuples) but exclude strings. Similarly, for the typing module, there doesn't seem to be a type for iterables that don't include strings.

Answer: As of March 2022, the answer is no. This issue has been discussed since at least July 2016. On a proposal to distinguish between str and Iterable[str], Guido van Rossum writes:  Since str is a valid iterable of str this is tricky. Various proposals have been made but they don't fit easily in the type system.  You'll need to list out all of the types that you want your functions to accept explicitly, using Union (pre-3.10) or | (3.10 and higher). e.g. For pre-3.10, use: from typing import Union ## Heading ## def operate_on_files(file_paths: Union[TypeOneName, TypeTwoName, etc.]) -> None: for path in file_paths: ...  For 3.10 and higher, use: ## Heading ## def operate_on_files(file_paths: TypeOneName | TypeTwoName | etc.) -> None: for path in file_paths: ...  If you happen to be using Pytype, it will not treat str as an Iterable[str] (as pointed out by Kelly Bundy). But, this behavior is typechecker-specific, and isn't widely supported in other typecheckers.
Question: Let's say I have a df pd.DataFrame( {'name':['pam','pam','bob','bob','pam','bob','pam','bob'], 'game_id':[0,0,1,1,0,2,1,2] } ) name game_id 0 pam 0 1 pam 0 2 bob 1 3 bob 1 4 pam 0 5 bob 2 6 pam 1 7 bob 2  I want to calculate how many games bob and amy have appeared in cumulatively. However, when I use .groupby() and .cumcount()+1, I get something different. I get a cumulative count within each game_id: df['games'] = df.groupby(['name','game_id']).cumcount()+1 name game_id games 0 pam 0 1 1 pam 0 2 2 bob 1 1 3 bob 1 2 4 pam 0 3 5 bob 2 1 6 pam 1 1 7 bob 2 2  When what I really want is a one total cumulative count rather than a cumulative count for each unique game_id. Here's an example of my desired output:  name game_id games 0 pam 0 1 1 pam 0 1 2 bob 1 1 3 bob 1 1 4 pam 0 1 5 bob 2 2 6 pam 1 2 7 bob 2 2  Note, in my actual dataset game_id is a random sequence of numbers.

Answer: Lets try sort df, check consecutive difference, create new group by cumsum and then resort the df new_df=df.sort_values(by=['name','game_id']) new_df=new_df.assign(rank=new_df['game_id']!=new_df['game_id'].shift()) new_df=new_df.assign(rank=new_df.groupby('name')['rank'].cumsum()).sort_index() print(new_df) name game_id rank 0 pam 0 1 1 pam 0 1 2 bob 1 1 3 bob 1 1 4 pam 0 1 5 bob 2 2 6 pam 1 2 7 bob 2 2
Question:  How to style the pandas dataframe as an excel table (alternate row colour)? Sample style:  Sample data: import pandas as pd import seaborn as sns df = sns.load_dataset("tips") 

Answer: If your final goal is to save to_excel, the only way to retain the styling after export is using the apply-based methods:  df.style.apply / df.style.applymap are the styling counterparts to df.apply / df.applymap and work analogously df.style.apply_index / df.style.applymap_index are the index styling counterparts (requires pandas 1.4.0+)  For the given sample, use df.style.apply to style each column with alternating row colors and df.style.applymap_index to style all row/col indexes: css_alt_rows = 'background-color: powderblue; color: black;' css_indexes = 'background-color: steelblue; color: white;' (df.style.apply(lambda col: np.where(col.index % 2, css_alt_rows, None)) # alternating rows .applymap_index(lambda _: css_indexes, axis=0) # row indexes (pandas 1.4.0+) .applymap_index(lambda _: css_indexes, axis=1) # col indexes (pandas 1.4.0+) ).to_excel('styled.xlsx', engine='openpyxl')    If you only care about the appearance in Jupyter, another option is to set properties for targeted selectors using df.style.set_table_styles (requires pandas 1.2.0+): # pandas 1.2.0+ df.style.set_table_styles([ {'selector': 'tr:nth-child(even)', 'props': css_alt_rows}, {'selector': 'th', 'props': css_indexes}, ])
Question: I have two dataframes and a rather tricky join to accomplish. The first dataframe: data = [[0, 'Standard1', [100, 101, 102]], [1, 'Standard2', [100, 102]], [2, 'Standard3', [103]]] df1 = pd.DataFrame(data, columns = ['RuleSetID', 'RuleSetName', 'KeyWordGroupID']) df1 Output: RuleSetID RuleSetName KeyWordGroupID 0 Standard1 [100, 101, 102] 1 Standard2 [100, 102] 2 Standard3 [103] ... ... ...  The second one: data = [[100, 'verahren', ['word1', 'word2']], [101, 'flaechen', ['word3']], [102, 'nutzung', ['word4', 'word5']], [103, 'ort', ['word6', 'word7']]] df2 = pd.DataFrame(data, columns = ['KeyWordGroupID', 'KeyWordGroupName', 'KeyWords']) df2 Output: KeyWordGroupID KeyWordGroupName KeyWords 100 verahren ['word1', 'word2'] 101 flaechen ['word3'] 102 nutzung ['word4', 'word5'] 103 ort ['word6', 'word7'] ... ... ...  The desired output: RuleSetID RuleSetName KeyWordGroupID 0 Standard1 [['word1', 'word2'], ['word3'], ['word4', 'word5']] 1 Standard2 [['word1', 'word2'], ['word4', 'word5']] 2 Standard3 [['word6', 'word7']]  I tried to convert the second dataframe into a dictionary using df.to_dict('records') and put it into a pandas apply user defined function to match via key values but it doesn't seem like a clean approach. Does someone has an approach to solve that? Any ideas are rewarded.

Answer: The main idea is to convert df2 as a dict mapping Series where the key is the KeyWordGroupID column and the value is the KeyWords column. You can use explode to flatten KeyWordGroupID column of df1 then map it to df2 then groupby to reshape your first dataframe: df1['KeyWordGroupID'] = ( df1['KeyWordGroupID'].explode().map(df2.set_index('KeyWordGroupID')['KeyWords']) .groupby(level=0).apply(list) ) print(df1) # Output RuleSetID RuleSetName KeyWordGroupID 0 0 Standard1 [[word1, word2], [word3], [word4, word5]] 1 1 Standard2 [[word1, word2], [word4, word5]] 2 2 Standard3 [[word6, word7]]
Question: I want to find a custom command in a project with many apps, how to get a list of all commands from all apps?

Answer: This command will list all the custom or existing command of all installed apps: python manage.py help
Question: a simple python name myapp.py:  import threading import os def app(environ, start_response): tid = threading.get_ident() pid = os.getpid() ppid = os.getppid() # ##### print('tid ================ ', tid) # why same tid? # ##### print('pid', pid) # print('ppid', ppid) # data = b"Hello, World!\n" start_response("200 OK", [ ("Content-Type", "text/plain"), ("Content-Length", str(len(data))) ]) return iter([data])  And I start with gunicorn: gunicorn -w 4 myapp:app [2022-03-28 21:59:57 +0800] [55107] [INFO] Starting gunicorn 20.1.0 [2022-03-28 21:59:57 +0800] [55107] [INFO] Listening at: http://127.0.0.1:8000 (55107) [2022-03-28 21:59:57 +0800] [55107] [INFO] Using worker: sync [2022-03-28 21:59:57 +0800] [55110] [INFO] Booting worker with pid: 55110 [2022-03-28 21:59:57 +0800] [55111] [INFO] Booting worker with pid: 55111 [2022-03-28 21:59:57 +0800] [55112] [INFO] Booting worker with pid: 55112 [2022-03-28 21:59:57 +0800] [55113] [INFO] Booting worker with pid: 55113  then I curl http://127.0.0.1:8000/ (or use a browser). logs below: tid ================ 4455738816 pid 55112 ppid 55107 tid ================ 4455738816 pid 55111 ppid 55107 tid ================ 4455738816 pid 55113 ppid 55107  the question is why the tid is same but the pid is not same. ps: the code is from https://gunicorn.org/ homepage.

Answer: Gunicorn creates multiple processes to avoid the Python GIL. Each process has a unique PID.  Regarding the threads, threading.get_ident() is a Python specific thread identifier, it should be regarded as meaningless and relevant only within the local process. Instead, you should use threading.get_native_id() which returns the unique system-wide thread identifier. Keep in mind the latter may be recycled and reused upon thread closure.
Question: I am a python pandas user but recently found about polars dataframe and it seems quite promising and blazingly fast. I am not able to find a way to open an excel file in polars. Polars is happily reading csv, json, etc. but not excel. I am extensive user of excel files in pandas and I want to try using polars. I have many sheets in excel that pandas automatically read. How can I do same with polars? What am I missing?

Answer: This is more of a workaround than a real answer, but you can read it into pandas and then convert it to a polars dataframe. import polars as pl import pandas as pd df = pd.read_excel(...) df_pl = pl.DataFrame(df)  You could, however, make a feature request to the Apache Arrow community to support excel files.
Question: I have an existing TypedDict containing multiple entries: from typing import TypedDict class Params(TypedDict): param1:str param2:str param3:str  I want to create the exact same TypedDict but with all the keys being optional so that the user can specify only certain parameters. I know I can do something like: class OptionalParams(TypedDict, total=False): param1:str param2:str param3:str  but the problem with this method is that I have to duplicate the code. Is there a way to inherit from Params by making the keys optional ? I tried to do class OptionalParams(Params, total=False): pass  but the linter does not understand that the parameters are optional

Answer: What you ask for is not possible - at least if you use mypy - as you can read in the comments of Why can a Final dictionary not be used as a literal in TypedDict? and on mypy's github: TypedDict keys reuse?. Pycharm seems to have the same limitation, as tested in the two other "Failed attempts" answers to your question. When trying to run this code: from typing import TypeDict params = {"a": str, "b": str} Params = TypedDict("Params", params)  mypy will give error: TypedDict() expects a dictionary literal as the second argument, thrown here in the source code.
Question: I have this code to create a date range dataframe dates_df = pd.date_range(start='01/01/2022', end='02/02/2022', freq='1H')  The problem is that the time is not UTC. It is dtype='datetime64[ns] instead of dtype='datetime64[ns, UTC] How can I generate the date range in UTC without having the generated time change?

Answer: pass the parameter timezone = 'utc' otherwise the result of date_range() is timezone neutral and can be interpreted as your desired timezone dates_df = pd.date_range(start='01/01/2022', end='02/02/2022', freq='1H',tz='UTC')  output: >>> DatetimeIndex(['2022-01-01 00:00:00+00:00', '2022-01-01 01:00:00+00:00', ... '2022-02-01 23:00:00+00:00', '2022-02-02 00:00:00+00:00'], dtype='datetime64[ns, UTC]', length=769, freq='H')
Question: Since Python 3.7, dictionaries are ordered. So why I can't get keys by index?

Answer: Building in such an API would be an "attractive nuisance": the implementation can't support it efficiently, so better not to tempt people into using an inappropriate data structure. It's for much the same reason that, e.g., a linked list rarely offers an indexing API. That's totally ordered too, but there's no efficient way to find the i'th element for an arbitrary i. You have to start at the beginning, and follow i links in turn to find the i'th. Same end result for a CPython dict. It doesn't use a linked list, but same thing in the end: it uses a flat vector under the covers, but basically any number of the vector's entries can be "holes". There's no way to jump over holes short of looking at each entry, one at a time. People expect a[i] to take O(1) (constant) time, not O(i) time.
Question: I understand that in python user-defined objects can be made callable by defining a __call__() method in the class definition. For example, class MyClass: def __init__(self): pass def __call__(self, input1): self.my_function(input1) def my_function(self, input1): print(f"MyClass - print {input1}") my_obj = MyClass() # same as calling my_obj.my_function("haha") my_obj("haha") # prints "MyClass - print haha"  I was looking at how pytorch makes the forward() method of a nn.Module object be called implicitly when the object is called and saw some syntax I didn't understand. In the line that supposedly defines the __call__ method the syntax used is, __call__ : Callable[..., Any] = _call_impl  This seemed like a combination of an annotation (keyword Callable[ following : ignored by python) and a value of _call_impl which we want to be called when __call__ is invoked, and my guess is that this is a shorthand for, def __call__(self, *args, **kwargs): return self._call_impl(*args, **kwargs)  but wanted to understand clearly how this method of defining functions worked. My question is: When would we want to use such a definition of callable attributes of a class instead of the usual def myfunc(self, *args, **kwargs)

Answer: Functions are normal first-class objects in python. The name to with which you define a function object, e.g. with a def statement, is not set in stone, any more than it would be for an int or list. Just as you can do a = [1, 2, 3] b = a  to access the elements of a through the name b, you can do the same with functions. In your first example, you could replace def __call__(self, input1): self.my_function(input1)  with the much simpler __call__ = my_function  You would need to put this line after the definition of my_function. The key differences between the two implementations is that def __call__(... creates a new function. __call__ = ... simply binds the name __call__ to the same object as my_function. The noticeable difference is that if you do __call__.__name__, the first version will show __call__, while the second will show my_function, since that's what gets assigned by a def statement.
Question: I'd like to sort the keys of the list_of_dicts according to the list_months. It works fine once I remove the digits (years) from the keys of list_of_dicts, but I cannot figure out how to use the regex correctly in the lambda function to include the digits. My code so far: import re list_months = ["Jan", "Feb", "Mar", "Apr", "May", "Jun", "Jul", "Aug", "Sep", "Oct", "Nov", "Dec"] list_of_dicts = [{'Apr23': '64.401'}, {'Aug23': '56.955'}, {'Dec23': '57.453'}, {'Feb23': '90.459'}, {'Jan23': '92.731'}, {'Jul23': '56.6'}, {'Jun23': '56.509'},{'Mar23': '86.209'}, {'May23': '58.705'}, {'Nov23': '57.368'}, {'Oct23': '56.711'}, {'Sep23': '57.952'}] r = re.compile("[a-zA-Z]{3}[0-9]{2}") print(sorted(list_of_dicts, key=lambda d: [k in d for k in list_months if re.search(r, k)], reverse=True)) 

Answer: No need for a regex here. dict_months = {m:i for i, m in enumerate(list_months)} result = sorted(list_of_dicts, key=lambda d: dict_months[next(iter(d))[:3]]) print(result) # [{'Jan23': '92.731'}, {'Feb23': '90.459'}, {'Mar23': '86.209'}, {'Apr23': '64.401'}, {'May23': '58.705'}, {'Jun23': '56.509'}, {'Jul23': '56.6'}, {'Aug23': '56.955'}, {'Sep23': '57.952'}, {'Oct23': '56.711'}, {'Nov23': '57.368'}, {'Dec23': '57.453'}]  If you also want to take the year into account, use def sortby(d): key = next(iter(d)) return int(key[3:]), dict_months[key[:3]] result = sorted(list_of_dicts, key=sortby)
Question: When I launch my discord.py bot with this code:  > from discord.ext import commands > > bot = commands.Bot(command_prefix = ",", description = "Bot de eagle57") > > bot.run("Mytoken")  I get this error: C:\Users\Elève\AppData\Local\Programs\Python\Python310\lib\site-packages\aiohttp\connector.py:964: RuntimeWarning: coroutine 'TCPConnector._resolve_host' was never awaited hosts = await asyncio.shield(self._resolve_host( RuntimeWarning: Enable tracemalloc to get the object allocation traceback Traceback (most recent call last): File "d:\Python\Bot_discord\main.py", line 5, in <module> bot.run("Mytoken") File "C:\Users\Elève\AppData\Local\Programs\Python\Python310\lib\site-packages\discord\client.py", line 723, in run return future.result() File "C:\Users\Elève\AppData\Local\Programs\Python\Python310\lib\site-packages\discord\client.py", line 702, in runner await self.start(*args, **kwargs) File "C:\Users\Elève\AppData\Local\Programs\Python\Python310\lib\site-packages\discord\client.py", line 665, in start await self.login(*args, bot=bot) File "C:\Users\Elève\AppData\Local\Programs\Python\Python310\lib\site-packages\discord\client.py", line 511, in login await self.http.static_login(token.strip(), bot=bot) File "C:\Users\Elève\AppData\Local\Programs\Python\Python310\lib\site-packages\discord\http.py", line 300, in static_login data = await self.request(Route('GET', '/users/@me')) File "C:\Users\Elève\AppData\Local\Programs\Python\Python310\lib\site-packages\discord\http.py", line 192, in request async with self.__session.request(method, url, **kwargs) as r: File "C:\Users\Elève\AppData\Local\Programs\Python\Python310\lib\site-packages\aiohttp\client.py", line 1012, in __aenter__ self._resp = await self._coro File "C:\Users\Elève\AppData\Local\Programs\Python\Python310\lib\site-packages\aiohttp\client.py", line 480, in _request conn = await self._connector.connect( File "C:\Users\Elève\AppData\Local\Programs\Python\Python310\lib\site-packages\aiohttp\connector.py", line 523, in connect proto = await self._create_connection(req, traces, timeout) File "C:\Users\Elève\AppData\Local\Programs\Python\Python310\lib\site-packages\aiohttp\connector.py", line 858, in _create_connection _, proto = await self._create_direct_connection( File "C:\Users\Elève\AppData\Local\Programs\Python\Python310\lib\site-packages\aiohttp\connector.py", line 964, in _create_direct_connection hosts = await asyncio.shield(self._resolve_host( TypeError: shield() got an unexpected keyword argument 'loop'  Does anyone have any idea why I have this error? I have already removed everything that is not required in the code and I have updated all my pip freeze, but the error doesn't change.

Answer: This is usually caused because of outdated aiohttp module You can run pip install -U aiohttp  and pip install -U discord.py  This will fix your issue in most cases
Question: There appear to be conflicting libraries of python that pip is trying to access, as you can see with the following error: [root@fedora user]# pip Traceback (most recent call last): File "/usr/local/bin/pip", line 5, in <module> from pip._internal import main File "/usr/local/lib/python3.10/site-packages/pip/_internal/__init__.py", line 40, in <module> from pip._internal.cli.autocompletion import autocomplete File "/usr/local/lib/python3.10/site-packages/pip/_internal/cli/autocompletion.py", line 8, in <module> from pip._internal.cli.main_parser import create_main_parser File "/usr/local/lib/python3.10/site-packages/pip/_internal/cli/main_parser.py", line 12, in <module> from pip._internal.commands import ( File "/usr/local/lib/python3.10/site-packages/pip/_internal/commands/__init__.py", line 6, in <module> from pip._internal.commands.completion import CompletionCommand File "/usr/local/lib/python3.10/site-packages/pip/_internal/commands/completion.py", line 6, in <module> from pip._internal.cli.base_command import Command File "/usr/local/lib/python3.10/site-packages/pip/_internal/cli/base_command.py", line 25, in <module> from pip._internal.index import PackageFinder File "/usr/local/lib/python3.10/site-packages/pip/_internal/index.py", line 14, in <module> from pip._vendor import html5lib, requests, six File "/usr/local/lib/python3.10/site-packages/pip/_vendor/html5lib/__init__.py", line 25, in <module> from .html5parser import HTMLParser, parse, parseFragment File "/usr/local/lib/python3.10/site-packages/pip/_vendor/html5lib/html5parser.py", line 8, in <module> from . import _tokenizer File "/usr/local/lib/python3.10/site-packages/pip/_vendor/html5lib/_tokenizer.py", line 16, in <module> from ._trie import Trie File "/usr/local/lib/python3.10/site-packages/pip/_vendor/html5lib/_trie/__init__.py", line 3, in <module> from .py import Trie as PyTrie File "/usr/local/lib/python3.10/site-packages/pip/_vendor/html5lib/_trie/py.py", line 6, in <module> from ._base import Trie as ABCTrie File "/usr/local/lib/python3.10/site-packages/pip/_vendor/html5lib/_trie/_base.py", line 3, in <module> from collections import Mapping ImportError: cannot import name 'Mapping' from 'collections' (/usr/lib64/python3.10/collections/__init__.py)  How can I fix this to be able to use pip? I have already tried dnf reinstall python and dnf reinstall python3 and dnf remove python3-pip, dnf install python3-pip.

Answer: I fixed this by removing all pip folders in /usr/local/lib/python3.10/site-packages