eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I am trying to create a network graph on correlation data and would like to color the nodes based on categories. Data sample view:  Data: import pandas as pd links_data = pd.read_csv("https://raw.githubusercontent.com/johnsnow09/network_graph/refs/heads/main/links_filtered.csv")  graph code: import networkx as nx G = nx.from_pandas_edgelist(links_data, 'var1', 'var2') # Plot the network: nx.draw(G, with_labels=True, node_color='orange', node_size=200, edge_color='black', linewidths=.5, font_size=2.5)   All the nodes in this network graph is colored as orange but I would like to color them based on Category variable. I have looked for more examples but not sure how to do it. I am also open to using other python libraries if required. Appreciate any help here !!

Answer: Since you have a unique relationship from var1 to Category, you could build a list of colors for all the nodes using: import matplotlib as mpl cmap = mpl.colormaps['Set3'].colors # this has 12 colors for 11 categories cat_colors = dict(zip(links_data['Category'].unique(), cmap)) colors = (links_data .drop_duplicates('var1').set_index('var1')['Category'] .map(cat_colors) .reindex(G.nodes) ) nx.draw(G, with_labels=True, node_color=colors, node_size=200, edge_color='black', linewidths=.5, font_size=2.5)  If you also want a legend: import matplotlib.patches as mpatches plt.legend(handles=[mpatches.Patch(color=c, label=label) for label, c in cat_colors.items()], bbox_to_anchor=(1, 1))  Output:
Question: With a set of data from an arbitrary set of lists (or dicts or other iter), I want to create a new list or tuple that has all the first entries, then all the 2nd, and so on, like an hstack. If I have a known set of data, I can zip them together like this: data = {'2015': [2, 1, 4, 3, 2, 4], '2016': [5, 3, 3, 2, 4, 6], '2017': [3, 2, 4, 4, 5, 3]} hstack = sum(zip(data['2015'], data['2016'], data['2017']), ()) print(hstack) # hstack: (2, 5, 3, 1, 3, 2, 4, 3, 4, 3, 2, 4, 2, 4, 5, 4, 6, 3)  But what if I don't know how many entries (or the keys) of the dict? For processing an arbitrary set of iterators, I tried: combined_lists = sum(zip(data[val] for val in data.keys()), ()) # combined_lists: ([2, 1, 4, 3, 2, 4], [5, 3, 3, 2, 4, 6], [3, 2, 4, 4, 5, 3])  And also: nums = sum(zip(num for val in data.keys() for num in data[val]), ()) # nums: (2, 1, 4, 3, 2, 4, 5, 3, 3, 2, 4, 6, 3, 2, 4, 4, 5, 3)  But both of these just keep the same order I could get from adding the sequences together. I was able to get it to work with: counts = [] entry = list(data.keys())[0] for idx, count in enumerate(data[entry]): for val in list(data.keys()): counts.append(data[val][idx]) # counts: [2, 5, 3, 1, 3, 2, 4, 3, 4, 3, 2, 4, 2, 4, 5, 4, 6, 3]  Which is fine, but it's a bit bulky. Seems like there should be a better way. Is there a way with list comprehension or some feature of zip that I missed? No imports preferred.

Answer: While @Tinyfold's answer works, it should be pointed out that using sum to concatenate a series of iterables is rather inefficient as noted in the documentation:  For some use cases, there are good alternatives to sum(). The preferred, fast way to concatenate a sequence of strings is by calling ''.join(sequence). To add floating-point values with extended precision, see math.fsum(). To concatenate a series of iterables, consider using itertools.chain().  This is because sum() at its core performs a series of additions, each of which is a binary operation that "adds" two operands into one value. This is fine for numeric values, but for tuples as in your example, each addition creates a new tuple by copying all items in tuples of both operands, so a series of additions of tuples becomes quadratic in time complexity. To keep the time complexity linear you can use itertools.chain instead as suggested by the documentation: from itertools import chain hstack = tuple(chain.from_iterable(zip(*data.values())))  or if you absolutely don't want any import (which should not be a problem because itertools is part of the standard library), use a generator expression: hstack = tuple(i for t in zip(*data.values()) for i in t)  Note that itertools.chain is faster than a generator expression/comprehension because the former is written purely in C while the latter has to be interpreted.
Question: The setup code: class MyContainer: def __init__(self): self.stuff = [1, 2, 3] def __iter__(self): print("__iter__") return iter(self.stuff) def __len__(self): print("__len__") return len(self.stuff) mc = MyContainer()  Now, in my shell: >>> i = iter(mc) __iter__ >>> [x for x in i] [1, 2, 3] >>> list(mc) __iter__ __len__ [1, 2, 3]  Why is __len__() getting called by list()? And where is that documented?

Answer: The behavior of calling __len__ of the given iterable during initialization of a new list is an implementation detail and is meant to help pre-allocate memory according to the estimated size of the result list, as opposed to naively and inefficiently grow the list as it is iteratively extended with items produced by a given generic iterable. You can find the logics in Objects/listobject.c of CPython, where it defaults the pre-allocation of memory to a size of 8 if the iterable has neither __len__ nor __length_hint__, which is documented in PEP-424: static int list_extend_iter_lock_held(PyListObject *self, PyObject *iterable) { PyObject *it = PyObject_GetIter(iterable); if (it == NULL) { return -1; } PyObject *(*iternext)(PyObject *) = *Py_TYPE(it)->tp_iternext; /* Guess a result list size. */ Py_ssize_t n = PyObject_LengthHint(iterable, 8); ... }
Question: Sets don't have a deterministic order in Python. Why then can you do tuple unpacking on a set in Python? To demonstrate the problem, take the following in CPython 3.10.12: a, b = {"foo", "bar"} # sets `a = "bar"`, `b = "foo"` a, b = {"foo", "baz"} # sets `a = "foo"`, `b = "baz"`  I recognize that the literal answer is that Python tuple unpacking works on any iterable. For example, you can do the following: def f(): yield 1 yield 2 a, b = f()  But why is there not a check used by tuple unpacking that the thing being unpacked has deterministic ordering?

Answer: The core "why" is: Because all features start at -100 points, and nobody thought it was worth preventing sets from being used in this context. Every new feature costs developer resources to write it, write tests for it, code review it, and then maintain it forever. There has to be a significant benefit to the feature to justify it. "Preventing people from doing something that is potentially useful in niche contexts to avoid (possibly accidental) misuse in other niche contexts" is essentially neutral on pros and cons. You could propose a feature that would enable this. If someone came up with a significant benefit that would not only cancel out the -100 points all features start at, but also cancel out the negative points applied because this would definitely break existing code in use right now, then they might deprecate (with a warning) iterable unpacking using sets and other unordered iterables, and in a year or three some new version of Python could eventually forbid it. I don't see it happening. Fundamentally:  It is useful for sets to be iterable (even if unordered iteration is bad in your opinion, sorted(someset) relies on being able to iterate sets to produce the list that it then sorts). So that's not going away. Iterable unpacking applies to all iterables; you'd need to special-case unordered iterables to explicitly block it. You'll never prevent all forms of the misuse you seem to dislike (something as simple as a, b = list(theset) will prevent the "misuse" from being detected) There are always going to be valid use cases this needlessly blocks, e.g. checking for a single element set by unpacking, [obj] = theset (with try/except used to handle when it's more than one), or processing elements in a destructive fashion one at a time, when you just need one arbitrary element at a time, e.g. first, *collection = collection (where collection starts as set but becomes a list as a side-effect here). Even if they put in an explicit means of detecting unordered iterables, e.g. an __unordered__ attribute on the class with C level support so it can be checked efficiently, that's still slowing down a highly optimized code path for little to no benefit.  So it's a feature that will never handle all cases of "misuse", slows down uses to prevent the misuse, breaks existing code, and is only an arguable benefit in the first place. So they haven't done it, and almost certainly never will do it.
Question: When I move the legend of a Seaborn plot from its default position to the top of the plot, I want to have the variable (hue) name in the same row as the possible variable values. Starting from this plot: import seaborn as sns penguins = sns.load_dataset("penguins") ax = sns.relplot(penguins, x="bill_length_mm",y="flipper_length_mm" , hue="species")   I want to move the legend on top of the plot(e.g. like this): ax = sns.relplot(penguins, x="bill_length_mm",y="flipper_length_mm" , hue="species") sns.move_legend(ax, "lower center", bbox_to_anchor=(.5, 1), ncol=4)   But I want the variable name (species in the example) to be left of its values instead of on top of them. When also setting the style argument, this works fine(ignoring that the legend is to wide in this example): ax = sns.relplot(penguins, x="bill_length_mm",y="flipper_length_mm" , hue="species", style="sex") sns.move_legend(ax, "lower center", bbox_to_anchor=(.5, 1), ncol=7)  

Answer: You could retrieve the legend, get its bbox and move the title down/left. Here I used half of the width of the bbox to the left and a fixed extra left and down: import seaborn as sns penguins = sns.load_dataset('penguins') g = sns.relplot(penguins, x='bill_length_mm', y='flipper_length_mm', hue='species') sns.move_legend(g, 'lower center', bbox_to_anchor=(0.5, 1), ncol=7) bbox = g.legend.get_window_extent() g.legend.get_title().set_position((-bbox.width/2-20, -20))  You might have to tweak a bit the fixed values (here 20/20), it's otherwise relatively stable to changes of the figure size. Here with 6x3 inches as an example:
Question: I am on Linux, created a Python 3.9 venv, installed ruff in the venv, wrote this code: def process_data(data: list[int]) -> str: match data: case []: return "No data" case [first, *_] if (average := lambda: sum(data) / len(data)) and average() > 50: return f"Data average is high: {average():.2f}, starting with {first}" case _: return f"Processed {len(data)} items."  The match syntax is not available in Python 3.9, so running ruff check, I would expect an error. I have tried to set project.requires-python and ruff.target-version but the latter seems to be used only for the formatter according to the docs. What am I missing?

Answer: Ruff doesn't support that (yet?). There's an open issue for precisely this problem.  Our parser doesn't take into account the Python version aka target-version setting while parsing the source code. This means that we would allow having a match statement when the target Python version is 3.9 or lower. We want to signal this to the user.
Question: Is there any way in polars to replace character just after the _ with uppercase using regex replace? So far I have achieved it using polars.Expr.map_elements. Is there any alternative using native expression API? import re import polars as pl # Initialize df = pl.DataFrame( { "id": [ "accessible_bidding_strategy.id", "accessible_bidding_strategy.name", "accessible_bidding_strategy.owner_customer_id", ] } ) # Transform df = df.with_columns( pl.col("id") .map_elements( lambda val: re.sub(r"_\w", lambda match: match.group(0)[1].upper(), val), return_dtype=pl.String, ) .alias("parsed_id") ) print(df)  Output shape: (3, 2) ┌───────────────────────────────────────────────┬───────────────────────────────────────────┐ │ id ┆ parsed_id │ │ --- ┆ --- │ │ str ┆ str │ ╞═══════════════════════════════════════════════╪═══════════════════════════════════════════╡ │ accessible_bidding_strategy.id ┆ accessibleBiddingStrategy.id │ │ accessible_bidding_strategy.name ┆ accessibleBiddingStrategy.name │ │ accessible_bidding_strategy.owner_customer_id ┆ accessibleBiddingStrategy.ownerCustomerId │ └───────────────────────────────────────────────┴───────────────────────────────────────────┘ 

Answer: I don't think it's possible to "dynamically" modify the replacement in any of the Polars replace methods. You could create all possible mappings and use .str.replace_many() import string pl.Config(fmt_table_cell_list_len=10, fmt_str_lengths=120) df.with_columns( pl.col("id").str.replace_many( [f"_{c}" for c in string.ascii_lowercase], [f"_{c}" for c in string.ascii_uppercase], ) .str.replace_all("_", "") .alias("parsed_id") )  shape: (3, 2) ┌───────────────────────────────────────────────┬───────────────────────────────────────────┐ │ id ┆ parsed_id │ │ --- ┆ --- │ │ str ┆ str │ ╞═══════════════════════════════════════════════╪═══════════════════════════════════════════╡ │ accessible_bidding_strategy.id ┆ accessibleBiddingStrategy.id │ │ accessible_bidding_strategy.name ┆ accessibleBiddingStrategy.name │ │ accessible_bidding_strategy.owner_customer_id ┆ accessibleBiddingStrategy.ownerCustomerId │ └───────────────────────────────────────────────┴───────────────────────────────────────────┘  Otherwise you'd probably need some form of .str.split() and .list.eval() df.with_columns( pl.col("id").str.split("_").list.eval( pl.element().first() + ( pl.element().slice(1).str.slice(0, 1).str.to_uppercase() + pl.element().slice(1).str.slice(1) ) .str.join() ) .list.first() .alias("parsed_id") )
Question: I'm trying to create a code to show me some stock stats. For that I need to iterate through a list of stocks in python and for each one show some details. So far I have this code: import yfinance as yf import pandas as pd tickerlist = ['AAPL', 'MSFT', 'AMZN'] stock_data = [] for stock in tickerlist: info = yf.Ticker(stock).info stock_data.append(info.get("symbol")) stock_data.append(info.get("bookValue"))) df = pd.DataFrame(stock_data) df  But it shows only one column with all data instead of one line per stock, with data headers.  I tweaked the code a little bit and I got one row per item but it repeats the last item only, instead of add each entry to row with its values. import yfinance as yf import pandas as pd tickerlist = ['AAPL', 'MSFT', 'AMZN'] for stock in tickerlist: info = yf.Ticker(stock).info df = pd.DataFrame({ 'symbol': info.get("symbol"), 'bookValue': info.get("bookValue")}, index=('0', '1', '2')) df  

Answer: Instead of adding each piece of info you need as a separate element to stock_data, you should have a list of iterables, where each element of the list contains all of the relevant data for that stock. Note that you can also explicitly provide the column names to get a "nicer" output: mport yfinance as yf import pandas as pd tickerlist = ['AAPL', 'MSFT', 'AMZN'] stock_data = [] for stock in tickerlist: info = yf.Ticker(stock).info stock_data.append((info.get("symbol"), info.get("bookValue"))) df = pd.DataFrame(stock_data, columns=("symbol", "bookValue")) print(df)  Output:  symbol bookValue 0 AAPL 3.767 1 MSFT 38.693 2 AMZN 24.655
Question: Trying to connect to posgresql base as Django wrote in its docs: https://docs.djangoproject.com/en/5.1/ref/databases/#postgresql-notes DATABASES = { "default": { "ENGINE": "django.db.backends.postgresql", "OPTIONS": { "service": "my_service", "passfile": ".my_pgpass", }, } }  I've created 2 files in my home directory .pg_service.conf [my_service] host=/var/run/postgresql user=dbms dbname=dbms_db port=5432  .pgpass /var/run/postgresql:5432:dbms_db:dbms:my_password  Such command as test of .pgpass:  psql -h localhost -U dbms dbms_db  is working. But the connect doesn't work: DATABASES = { "default": { "ENGINE": "django.db.backends.postgresql", "OPTIONS": { "service": "my_service", "passfile": ".pgpass", }, } }  with such error  Traceback (most recent call last): File "/home/www/projects/amodulesu/venv/lib/python3.11/site-packages/django/db/backends/base/base.py", line 279, in ensure_connection self.connect() File "/home/www/projects/amodulesu/venv/lib/python3.11/site-packages/django/utils/asyncio.py", line 26, in inner return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/home/www/projects/amodulesu/venv/lib/python3.11/site-packages/django/db/backends/base/base.py", line 256, in connect self.connection = self.get_new_connection(conn_params) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/www/projects/amodulesu/venv/lib/python3.11/site-packages/django/utils/asyncio.py", line 26, in inner return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/home/www/projects/amodulesu/venv/lib/python3.11/site-packages/django/db/backends/postgresql/base.py", line 332, in get_new_connection connection = self.Database.connect(**conn_params) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/www/projects/amodulesu/venv/lib/python3.11/site-packages/psycopg/connection.py", line 119, in connect raise last_ex.with_traceback(None) psycopg.OperationalError: connection failed: connection to server on socket "/var/run/postgresql/.s.PGSQL.5432" failed: FATAL: Peer authentication failed for user "dbms" ... File "/home/www/projects/amodulesu/venv/lib/python3.11/site-packages/psycopg/connection.py", line 119, in connect raise last_ex.with_traceback(None) django.db.utils.OperationalError: connection failed: connection to server on socket "/var/run/postgresql/.s.PGSQL.5432" failed: FATAL: Peer authentication failed for user "dbms"  whats wrong with my code? Trying to use export vars in debian - but it doesnt work too.. Here is pg_hba.conf 

Answer: Cant use  'service': 'my_service', 'passfile': '.pgpass',  So use decouple and tradional DATABASES = { 'default': { 'ENGINE': 'django.db.backends.postgresql', 'NAME': 'dbms_db', 'USER': 'dbms', 'PASSWORD': DB_PASS, 'HOST': '127.0.0.1', 'PORT': '5432', } }
Question: I have a simple python script and I want to enable dragging and dropping files into the python script in Windows 11. This works. But only if I set python launcher as default application for python file types. I want to launch my editor when double clicking. My idea is to create a batch file as default application for the python script and launch my editor if no parameters are passed and launch it with python launcher if it has parameters. I cannot get it to fully work. But when opening the editor command window is created that doesn't close automatically. test drag.py: import sys import os if __name__ == "__main__": test_file_name = os.path.join(os.path.split(sys.argv[0])[0], "test.txt") if len(sys.argv) > 1: with open(test_file_name, 'w', newline='\n') as f: f.write("file names:") f.write("\n") for file_path in sys.argv[1:]: f.write(file_path) f.write("\n") else: with open(test_file_name, 'w', newline='\n') as f: f.write("no file names") f.write("\n")  C:\drag\python_drag.cmd: @echo off if not "%~2"=="" ( :: Arguments detected, so files were dragged on it. Calling it with python. START /B python.exe %* ) else ( :: No arguments detected, so double clicked. Opening it with editor (vscode) START /B code %1 )  Open command key is updated via command line: reg add "HKEY_CLASSES_ROOT\Python.File\shell\open\command" /ve /t REG_EXPAND_SZ /d "\"C:\drag\python_drag.cmd\" \"%L\" %*" /f  Set drophandler if needed (if installing latest python version with py launcher it should be set). enable_python_drag_and_drop.reg: Windows Registry Editor Version 5.00 [HKEY_CLASSES_ROOT\Python.File\shellex\DropHandler] @="{60254CA5-953B-11CF-8C96-00AA00B8708C}"  When starting python a window briefly opens (you can only see the outline) and then it closes automaticallt. When starting VSCODE it opens an additional window that doesn't close. How to fix it? Edit: I figured out how to at least minimize the window: START /w /min cmd /c "call code %1"  or: START /w /min cmd /c "call code.cmd %1" 

Answer: code.exe has been moved out of the bin folder. Inside the bin folder is only code and code.cmd. It is now one folder up. Running code.exe directly solves the problem. No window is created. Here is C:\drag\python_drag.cmd: @echo off if not "%~2"=="" ( rem Arguments detected, so files were dragged on it. Calling it with python. START "Python Script Execution" python.exe %* ) else ( rem No arguments detected, so double clicked. Opening it with editor (vscode) START "" "%LOCALAPPDATA%\Programs\Microsoft VS Code\code.exe" %1 )  And it's added with this command: reg add "HKEY_CLASSES_ROOT\Python.File\shell\open\command" /ve /t REG_SZ /d "\"C:\drag\python_drag.cmd\" \"%L\" %*" /f
Question: I use python / imagehash.phash to search for photos similar to a given one, i.e. photos so that the hamming distance is zero. Now I also would like to detect photos that are flipped or rotated copies - the hamming distance is > 0. Instead of flipping or rotating the photo and then calculating an new pHash I would like to derive the pHash of a rotated / flipped photo directly from the original pHash. My idea is to convert the hash into an numpy array, use np.flip etc and convert the array back into a pHash. Questions:  How can I convert the hash into an numpy array and back? is there a better way to search for rotated / flipped images given the hash of the original photo? 

Answer: From reading the source code, I found that you can convert between an imagehash.ImageHash object and a binary array using .hash and the imagehash.ImageHash() constructor. from PIL import Image import imagehash img = Image.open("house.jpg") image_hash_obj = imagehash.phash(img) print(image_hash_obj) hash_array = image_hash_obj.hash print(hash_array) hash_array[0, 0] = False image_hash_obj2 = imagehash.ImageHash(hash_array) print(image_hash_obj2)  Output: fde1921f6c20b34c [[ True True True True True True False True] [ True True True False False False False True] [ True False False True False False True False] [False False False True True True True True] [False True True False True True False False] [False False True False False False False False] [ True False True True False False True True] [False True False False True True False False]] 7de1921f6c20b34c
Question: I am trying to access the Python Client V4 for DyDx from within a scala project. I integrated the previous V3 using scalapy library. But V4 contains asynchronous calls which I don't know how I should handle. So for example the short term order composite example's test function begins with the following lines: node = await NodeClient.connect(TESTNET.node) indexer = IndexerClient(TESTNET.rest_indexer)  I can access the connect call as follows in my scala code:  val network = py.module("dydx_v4_client.network") val client = py.module("dydx_v4_client.node.client")  If I print out the value for client it gives me  <coroutine object NodeClient.connect at 0x7fe18e87a340>  The connect call in NodeClient is defined as asynchronous: async def connect(config: NodeConfig) -> Self: ...  How can I actually execute this coroutine object?

Answer: A solution I found is to use the asyncio.run function which can be used to run coroutine:  val asyncio = py.module("asyncio") val node = asyncio.run(client.NodeClient.connect(network.TESTNET.node))