eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I would like to create a NetworkX graph and visualize it using d3js similarly to the javascript example in the NetworkX docs. This graph is very similar to the interactive graph on the NetworkX homepage page. The example works for me, but I would like to add hyperlinks to the nodes. I think I have node attributes called "xlink:href", but I have not figured out how. This question was ansered for NewtworkX and visualization with bokeh here. I have not tested this example, since I want to use d3js. The code below is available here. So far: import json import networkx as nx G = nx.Graph() G.add_node('Node1') G.add_node('Node2') G.add_edge('Node1', 'Node2') for n in G: G.nodes[n]["name"] = "My" + str(n) G.nodes[n]["xlink:href"] = "http://google.com" # <==<< link Not working d = nx.json_graph.node_link_data(G) json.dump(d, open("force/force.json", "w")) print("Wrote node-link JSON data to force/force.json")  The above produces: {'directed': False, 'multigraph': False, 'graph': {}, 'nodes': [{'name': 'MyNode1', 'xlink:href': 'http://google.com', 'id': 'Node1'}, {'name': 'MyNode2', 'xlink:href': 'http://google.com', 'id': 'Node2'}], 'links': [{'source': 'Node1', 'target': 'Node2'}]}  Which can be visualized like this: # Serve the file over http to allow for cross origin requests import flask app = flask.Flask(__name__, static_folder="force") @app.route("/") def static_proxy(): return app.send_static_file("force.html") app.run(port=8001)   Interestingly, the tooltip on the graph displays "Node2" and not "MyNode2". Links collected while trying to solve this:  https://github.com/simonlindgren/nXd3 http://www.d3noob.org/2014/05/including-html-link-in-d3js-tool-tip.html https://networkx.org/documentation/stable/reference/readwrite/json_graph.html 

Answer: Here's a solution using gravis, which accepts graph objects from NetworkX, iGraph, graph-tool and some other Python libraries and can visualize them with either d3.js, vis.js or three.js with a single function call. Disclaimer: I'm the developer of gravis. Your use case appears to be 1) creating an interactive graph visualization with d3 and 2) serve it via a web server such as Flask. Since this was part of the motivation to build gravis, I think it might fit well here. import gravis as gv import networkx as nx g = nx.Graph() g.add_node('Node1') g.add_node('Node2') g.add_edge('Node1', 'Node2') for n in g: g.nodes[n]["name"] = f"My {n}" g.nodes[n]["hover"] = '<a href="http://google.com">Google</a>' fig = gv.d3(g, node_label_data_source='name') fig.display()  The last line opens a browser window and shows the visualization. Alternatively you can also use fig.to_html() to get a standalone HTML text that you can serve via a web server like Flask. The hyperlink is shown when hovering over a node and can easily be clicked since the pop up disappears with a time delay. Yet another use case that can easily be serve is creating embedded graph visualizations in a Jupyter notebook, which may help in prototyping your app:  (The mouse cursor was hidden from the screenshot. It's hovering over node 1, hence the pop up with the hyperlink is visible.)
Question: My current poetry.lock file looks like this (part of it) # This file is automatically @generated by Poetry and should not be changed by hand. [[package]] name = "annotated-types" version = "0.6.0" description = "Reusable constraint types to use with typing.Annotated" category = "main" optional = false python-versions = ">=3.8" files = [ {file = "annotated_types-0.6.0-py3-none-any.whl", hash = "sha256:0641064de18ba7a25dee8f96403ebc39113d0cb953a01429249d5c7564666a43"}, {file = "annotated_types-0.6.0.tar.gz", hash = "sha256:563339e807e53ffd9c267e99fc6d9ea23eb8443c08f112651963e24e22f84a5d"}, ] [[package]] name = "asgiref" version = "3.7.2" description = "ASGI specs, helper code, and adapters" category = "main" optional = false python-versions = ">=3.7" files = [ {file = "asgiref-3.7.2-py3-none-any.whl", hash = "sha256:89b2ef2247e3b562a16eef663bc0e2e703ec6468e2fa8a5cd61cd449786d4f6e"}, {file = "asgiref-3.7.2.tar.gz", hash = "sha256:9e0ce3aa93a819ba5b45120216b23878cf6e8525eb3848653452b4192b92afed"}, ]  For every package there is a category Now I need to add an extra package. I run poetry add --group main beautifulsoup4 , but category is removed from each line. What should I do to: 1) install a new package 2) make category fields stay the same

Answer: category was removed since poetry 1.5, that is why it's removed in poetry.lock file after any changes
Question: Why does hatch not save the virtual environment in the same directory I am working in? I went through the doc and couldn't find a way to achieve that. Maybe it makes sense that it's that way but I can't really think why. Is there a way to create it in the project directory? Or a good reason to not want to do that?

Answer: When you use a tool like hatch, you are basically giving up precise control over the virtual environment in favor of letting hatch manage it for you; this includes letting hatch decide where to keep all such managed environments. That said, hatch stores virtual environments in a specific data directory that you can specify using the --data-dir option to the hatch command itself. $ mkdir hatchtest; cd hatchtest $ ls $ hatch --data-dir . env create [...] $ ls env  (The default data directory likely varies from platform to platform; it's ~/Library/Application Support/hatch on my macOS box.)
Question: I want to know what the technical term for a colon in the context of introducing a suite after a statement is called. I do not mean colons in slices, key-value pairs or type hints, but this kind of usage: if True: pass else: pass try: raise except: pass def foo(): pass lambda: ... # I don't know if this is appertaining  I did not really find a fitting term online. Most only call them colons, while expecting the reader to understand the meaning of a colon in this context in Python. I found this website which uses the term "colon operator", but that doesn't feel right to me - especially as it lumps together all uses of colons under that umbrella term. I apologise if I have missed an obvious answer somewhere.

Answer: It's simply called a "colon" :, but I might say and the grammar spec suggests you'd say something like "the start of a block" https://docs.python.org/3/reference/grammar.html  # COMPOUND STATEMENTS # =================== # Common elements # --------------- block: | NEWLINE INDENT statements DEDENT | simple_stmts
Question: My input is a list: data = [ -1, 0, 'a','b', 1, 2, 3, 'c', 6, 'd', 'e', .4, .5, 'a', 'b', 4, 'f', 'g', ]  I'm trying to form groups (dictionary) where the keys are the strings and the values are the numbers right after them. There are however three details I should consider:  The list of data I receive can sometimes have leading non-string values that should be ignored The number of strings for each group is variable but the minimum is always 1 Some groups can appear multiple times (example: a/b)  For all of that I made the code below: start = list(map(type, data)).index(str) wanted = {} for i in data[start:]: strings = [] if type(i) == str: strings.append(i) numbers = [] else: numbers.append(i) wanted['/'.join(strings)] = numbers  This gives me nearly what I'm looking for: {'a': [], 'b': [4], '': [4], 'c': [6], 'd': [], 'e': [0.4, 0.5], 'f': [], 'g': []}  Can you show me how to fix my code? My expected output is this: {'a/b': [1, 2, 3, 4], 'c': [6], 'd/e': [0.4, 0.5], 'f/g': []} 

Answer: You can use itertools.groupby with a key function that tests if the current item is a string. To skip possible leading non-string items, fetch the first group, and fetch the next group again as a replacement if the first group items are not strings. For each group of items, if they are strings, join them as a key; otherwise extend the list under that key with the items. If the last group of items are strings, set it to an empty list as a default: from itertools import groupby, chain output = {} groups = groupby(data, lambda i: isinstance(i, str)) try: is_str, group = next(groups) if not is_str: is_str, group = next(groups) except StopIteration: pass else: for is_str, group in chain([(is_str, group)], groups): if is_str: key = '/'.join(group) else: output.setdefault(key, []).extend(group) if is_str: output.setdefault(key, [])  output becomes: {'a/b': [1, 2, 3, 4], 'c': [6], 'd/e': [0.4, 0.5], 'f/g': []}  Demo: https://ideone.com/rcOSzV
Question: Consider the model below class Faculty(models.Model): name = models.CharField(max_length=255) short_name = models.CharField(max_length=15) description = models.CharField(max_length=512) logo = models.ImageField(upload_to=faculty_upload_to, null=True)  That I do Faculty.objects.create() or faculty = Faculty() faculty.save()  this creates an empty entry in the database >>> from universities.models import * >>> Faculty.objects.create() <Faculty: Faculty object (2)>  Why doesn't Django give me an integrity error? I am using Django 5.0.

Answer: CharFields default to an empty string (""), which is a valid value for a VARCHAR column. You may consider adding a CheckConstraint if you want to validate values at the database level: class Faculty(models.Model): name = models.CharField(max_length=255) short_name = models.CharField(max_length=15) description = models.CharField(max_length=512) logo = models.ImageField(upload_to=faculty_upload_to, null=True) class Meta: constraints = [ models.CheckConstraint(check=~Q(name="")), models.CheckConstraint(check=~Q(short_name="")), models.CheckConstraint(check=~Q(description="")), ]
Question: This is my dataframe: import pandas as pd df = pd.DataFrame( { 'a': [110, 115, 112, 180, 150, 175, 160, 145, 200, 205, 208, 203, 206, 207, 208, 209, 210, 215], 'b': [1, 1, 0, 1, 0, 1, 0, 0, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1], } )  And this is the output that I want. I want to create column c.  a b c 0 110 1 110 1 115 1 115 2 112 0 115 3 180 1 180 4 150 0 180 5 175 1 180 6 160 0 180 7 145 0 180 8 200 1 200 9 205 1 205 10 208 1 208 11 203 0 208 12 206 1 208 13 207 1 208 14 208 1 208 15 209 1 209 16 210 1 210 17 215 1 215  When df.a > df.a.shift(1) b is 1 otherwise it is 0. Steps needed: a) Find where the streak of 1 in b ends. b) Keep the maximum value of the streak. c) Put that value in c until a greater value is found in a. For example when 180 is found in b: a) Row 3 has streak of 1. b) Maximum value of the streak is 180. c) df.c = 180 until a greater value is found in a. In this case it is 200 at row 8. It was not easy to elaborate the problem. Maybe I have described the problem with wrong words. So If there are any questions feel free to ask in the comments. And I really appreciate if you introduce a built-in way or a clean way to create column b. I put those 1 and 0s manually. This is what I have tried. But it does not feel like a correct approach. df['streak'] = df['b'].ne(df['b'].shift()).cumsum() df['max'] = df.groupby('streak')['a'].max() 

Answer: You just want cummax: df['c'] = df['a'].cummax()  Output:  a b c 0 110 1 110 1 115 1 115 2 112 0 115 3 180 1 180 4 150 0 180 5 175 1 180 6 160 0 180 7 145 0 180 8 200 1 200 9 205 1 205 10 208 1 208 11 203 0 208 12 206 1 208 13 207 1 208 14 208 1 208 15 209 1 209 16 210 1 210 17 215 1 215
Question: I have not found a way to do this as the BIT data type in sqlalchemy inherits from Boolean datatype.  flags = Column(BIT(4), nullable=False) # BIT column with length 4 sql: `flags` BIT(4) NOT NULL DEFAULT b'0',  AttributeError: 'BIT' object has no attribute 'length'

Answer: BIT is not a standard data type (neither in SQL Standard nor in SQLAlchemy) but it is supported by e.g. PostgreSQL, SQL Server, MariaDB and MySQL. As @danblack already mentioned in his comment, BIT type is defined in types.py (but also in base.py as MSBit). That means you have to use types of your dialect, in this case mysql (mariadb is also stored under dialects/mysql). e.g. from sqlalchemy.dialects.mysql.types import BIT .... table_def= Table('my_table', meta, Column('flags', BIT(4), nullable=False))
Question: I have this 2d numpy matrix a = np.array( [[ 1 2 3 4 5] [ 6 7 8 9 10] [11 12 13 14 15] [16 17 18 19 20] [21 22 23 24 25] [26 27 28 29 30]])  and I want to extract these values from it just for the sake of learning :) [[11,12],[16,17],[29,30]]  After so many tries I ended up in chatGPT which gave me a wrong answer :(. chatGPT suggested this a[[2, 3, 5], [0, 1, 1]] but yeilded these values [11 17 27]. Any help would be appreciated thanks

Answer: You need to add commas to your array. You can use numpy indexing: https://numpy.org/doc/stable/user/basics.indexing.html import numpy as np a=np.array([[ 1, 2, 3, 4, 5], [ 6, 7, 8, 9, 10], [11, 12, 13, 14, 15], [16, 17, 18, 19, 20], [21, 22, 23, 24, 25], [26, 27, 28, 29, 30]])   print(a[2:4,0:2])  or print(a[[2,3],0:2])  #output array([[11, 12], [16, 17]])   print(a[5:,3:])  or print(a[[5],3:])  #output array([[29, 30]])  Everything in one go: a[[2,3],0:2].tolist() + a[[5],3:].tolist()  or a[2:4,0:2].tolist() + a[5:,3:].tolist()  #output [[11, 12], [16, 17], [29, 30]]
Question: I have a multiple rest api endpoints (urls) which streams data. I wonder what would be the best approach to read from all of them in one / many processes. currently I'm reading the data only from one url, doing something like:  s = requests.Session() resp = s.get(url, headers=headers, stream=True) for line in resp.iter_lines(): if line: print(line)  I would like to do the same with more urls and I wonder what would be the best approach here.

Answer: Here is an example how you can read from multiple URLs using concurrent.futures.ThreadPoolExecutor. But this is just one approach, you can use multiprocessing, asyncio/aiohttp, etc. from concurrent.futures import ThreadPoolExecutor import requests def get_from_api(tpl): session, url = tpl resp = session.get(url, stream=True) # just for example: count_lines = 0 for line in resp.iter_lines(): count_lines += 1 return url, count_lines def main(): api_urls = [ "https://google.com", "https://yahoo.com", "https://facebook.com", "https://instagram.com", # ...etc. ] with ThreadPoolExecutor(max_workers=2) as pool, requests.session() as session: for url, count_lines in pool.map( get_from_api, ((session, url) for url in api_urls) ): print(url, count_lines) if __name__ == "__main__": main()  Prints: https://google.com 17 https://yahoo.com 648 https://facebook.com 26 https://instagram.com 50   EDIT: Using asyncio/aiohttp: import asyncio # Streaming API: # https://docs.aiohttp.org/en/stable/streams.html#streaming-api import aiohttp async def fetch(session, url): while True: async with session.get(url) as response: reader = response.content cnt = 0 async for line in reader: cnt += 1 print(f"{url}: {cnt} lines read") await asyncio.sleep(3) async def main(): urls = [ "https://google.com", # Replace with actual URLs "https://facebook.com", ] async with aiohttp.ClientSession() as session: tasks = {asyncio.create_task(fetch(session, url)) for url in urls} # this loops indifinitely: await asyncio.gather(*tasks) if __name__ == "__main__": asyncio.run(main())  Prints: https://google.com: 17 lines read https://facebook.com: 26 lines read https://google.com: 655 lines read https://facebook.com: 26 lines read ...
Question: I have the following xml file: <?xml version="1.0" encoding="utf-8"?> <components version="1.0.0"> <component type="foo"> <maintag> <subtag> <check>Foo</check> </subtag> <subtag> <check></check> </subtag> <subtag> </subtag> </maintag> </component> </components>  I want to check that each subtag element has child element check that has non-empty value. It should print an error if:  check is present but is empty check is not present at all in one or more subtag  How would I do that? I came up with this but it doesn't do exactly what I want from lxml import etree # type: ignore def parse_xml(path: str) -> list: root = etree.parse(path) components = root.xpath("/components/component") return list(components) path = "test.xml" for p in parse_xml(path)[0].iter('check'): if not len(str(p)) > 0: print("check tag empty") 

Answer: You can translate your two conditions to an XPath : root = etree.parse("test.xml") expr = "//subtag[not(check/text()) or not(check)]" not any(e is not None for e in root.xpath(expr)) # False  Alternatively, if you need to verbose the checks, you can do something like : for idx, st in enumerate(root.xpath("//subtag"), 1): if (check:=st.find("check")) is None: print(f"'check' at 'subtag' {idx} is absent") elif not check.text: print(f"'check' at 'subtag' {idx} is empty") else: ...  Output : # 'check' at 'subtag' 2 is empty # 'check' at 'subtag' 3 is absent
Question: When pushing two plots together, I'd like to have the the ticks set to 'inout' and have it properly displayed over both the charts, but the 2nd chart is overlapping it.  import matplotlib.pyplot as plt import numpy as np t = np.arange(0.01, 5.0, 0.01) s1 = np.sin(2 * np.pi * t) s2 = np.exp(-t) ax1 = plt.subplot(211) plt.plot(t, s1) plt.tick_params('x', labelbottom=False, direction='inout', length=6) ax2 = plt.subplot(212, sharex=ax1) plt.plot(t, s2) plt.subplots_adjust(hspace=0) plt.show()  We can adjust the hspace and see that they're there, but just getting covered.  I've tried changing the z-order, but that didn't work. plt.tick_params('x', labelbottom=False, direction='inout', length=6, zorder=100) 

Answer: IIUC, you can create ax2 before ax1: import matplotlib.pyplot as plt import numpy as np t = np.arange(0.01, 5.0, 0.01) s1 = np.sin(2 * np.pi * t) s2 = np.exp(-t) ax2 = plt.subplot(212) plt.plot(t, s2) ax1 = plt.subplot(211, sharex=ax2) plt.plot(t, s1) plt.tick_params('x', labelbottom=False, direction='inout', length=6) plt.subplots_adjust(hspace=0) plt.show()  Output:
Question: I need to run regression for each group, then pass the coefficients into the new column b. Here is my code: Self-defined function: def simplereg(g, y, x): try: xvar = sm.add_constant(g[x]) yvar = g[y] model = sm.OLS(yvar, xvar, missing='drop').fit() b = model.params[x] return pd.Series([b*100]*len(g)) except Exception as e: return pd.Series([np.NaN]*len(g))  Create sample data: import pandas as pd import numpy as np # Setting the parameters gvkeys = ['A', 'B', 'C', 'D'] # Possible values for gvkey years = np.arange(2000, 2020) # Possible values for year # Number of rows for each gvkey, ensuring 5-7 observations for each num_rows_per_gvkey = np.random.randint(5, 8, size=len(gvkeys)) total_rows = sum(num_rows_per_gvkey) # Creating the DataFrame np.random.seed(0) # For reproducibility df = pd.DataFrame({ 'gvkey': np.repeat(gvkeys, num_rows_per_gvkey), 'year': np.random.choice(years, size=total_rows), 'y': np.random.rand(total_rows), 'x': np.random.rand(total_rows) }) df.sort_values(by='year', ignore_index=True, inplace=True) # make sure if the code can handle even data without sort  Run groupby code: df['b'] = df.groupby('gvkey').apply(simplereg, y='y', x='x')  However, the code return column 'b' with all N/A May I ask where is the issue and how to fix it? Thank you

Answer: Is it a bad idea to catch all exceptions? Obviously yes. In addition you do not display any error message. If your function returns NaN values, it's probably because your code is throwing an exception. Your code works well for me if I import statsmodels.api as sm and make minor changes: import statsmodels.api as sm def simplereg(g, y, x): try: xvar = sm.add_constant(g[x]) yvar = g[y] model = sm.OLS(yvar, xvar, missing='drop').fit() b = model.params[x] return pd.Series([b*100]*len(g), index=g.index) # reindex here except Exception as e: print(e) # At least, print exception here return pd.Series([np.NaN]*len(g), index=g.index) # reindex here # drop the first group level (gvkey) to align indexes df['b'] = df.groupby('gvkey').apply(simplereg, y='y', x='x').droplevel('gvkey')  Output: >>> df gvkey year y x b 0 A 2000 0.799159 0.128926 -55.326856 1 B 2001 0.774234 0.253292 -68.351309 2 A 2003 0.461479 0.315428 -55.326856 3 A 2003 0.780529 0.363711 -55.326856 4 B 2004 0.521848 0.208877 -68.351309 5 C 2005 0.612096 0.656330 6.342994 6 D 2005 0.060225 0.096098 36.320231 7 B 2006 0.414662 0.161310 -68.351309 8 C 2006 0.456150 0.466311 6.342994 9 A 2007 0.118274 0.570197 -55.326856 10 C 2007 0.568434 0.244426 6.342994 11 C 2008 0.943748 0.196582 6.342994 12 D 2009 0.681820 0.368725 36.320231 13 B 2009 0.639921 0.438602 -68.351309 14 A 2012 0.870012 0.670638 -55.326856 15 B 2012 0.264556 0.653108 -68.351309 16 C 2013 0.616934 0.138183 6.342994 17 C 2014 0.018790 0.158970 6.342994 18 A 2015 0.978618 0.210383 -55.326856 19 D 2015 0.666767 0.976459 36.320231 20 D 2016 0.437032 0.097101 36.320231 21 C 2017 0.617635 0.110375 6.342994 22 B 2018 0.944669 0.102045 -68.351309 23 B 2019 0.143353 0.988374 -68.351309 24 D 2019 0.359508 0.820993 36.320231 25 D 2019 0.697631 0.837945 36.320231
Question: All the dfs have a key col "id". pd.merge is not a viable option even with the suffix option. There are over 40k cols in each of the dfs so column binding and deleting later (suffix_x) is not an option. Exactly 50k (common) rows in each of the dfs identified by "id" col. Minimal example with two common cols: df1 = pd.DataFrame({ 'id': ['a', 'b', 'c'], 'col1': [123, 121, 111], 'col2': [456, 454, 444], 'col3': [786, 787, 777], }) df2 = pd.DataFrame({ 'id': ['a', 'b', 'c'], 'col1': [123, 121, 111], 'col2': [456, 454, 444], 'col4': [11, 44, 77], }) df3 = pd.DataFrame({ 'id': ['a', 'b', 'c'], 'col1': [123, 121, 111], 'col2': [456, 454, 444], 'col5': [1786, 1787, 1777], })  Final answer: finaldf = pd.DataFrame({ 'id': ['a', 'b', 'c'], 'col1': [123, 121, 111], 'col2': [456, 454, 444], 'col3': [786, 787, 777], 'col4': [11, 44, 77], 'col5': [1786, 1787, 1777], }) 

Answer: If memory is limiting and the dataframes are already aligned, you could try to set up the output and update it: from functools import reduce dfs = [df1, df2, df3] cols = reduce(lambda a,b: a.union(b, sort=False), (x.columns for x in dfs)) out = pd.DataFrame(index=dfs[0].index, columns=cols) for x in dfs: out.update(x)  Variant for the last step: out = pd.DataFrame(dfs[0], columns=cols) for x in dfs[1:]: out.update(x)  Output:  id col1 col2 col3 col4 col5 0 a 123 456 786 11 1786 1 b 121 454 787 44 1787 2 c 111 444 777 77 1777
Question: This is my DataFrame: import pandas as pd df = pd.DataFrame( { 'a': [101, 90, 11, 120, 1] } )  And this is the output that I want. I want to create column y:  a y 0 101 101.0 1 90 101.0 2 11 90.0 3 120 120.0 4 1 120.0  Basically, values in a are compared with their previous value, and the greater one is selected. For example for row 1, 90 is compared with 101. 101 is greater so it is selected. I have done it in this way: df['x'] = df.a.shift(1) df['y'] = df[['a', 'x']].max(axis=1)  Is there a cleaner or some kind of built-in way to do it?

Answer: You can use np.fmax to get the maxima without creating an additional column: df["y"] = np.fmax(df["a"], df["a"].shift(1))  This outputs:  a y 0 101 101.0 1 90 101.0 2 11 90.0 3 120 120.0 4 1 120.0  We use np.fmax() to ignore the NaN created when shifting df["a"].
Question: In pandas dataframe, I'm totally confused of how to use the method of groupby() over consecutive duplicates by sum values in column Let's say I have the following DataFrame df : index type value 0 profit 11 1 profit 10 2 loss -5 3 profit 50 4 profit 15 5 loss -30 6 loss -25 7 loss -10  what I'm looking to is: index type grand 0 profit 21 # total of 11 + 10 = 21 1 loss -5 # the same value as this row NOT consecutive duplicated 2 profit 65 # total of 50 + 15 = 65 3 loss -65 # total of -30 -25 -10 = -65  What I tried to do:  df['grand'] = df.groupby(df['type'].ne(df['type'].shift()).cumsum()).cumcount()  but it gives me counting the consecutive duplicated I tried to iterate through the rows with several solutions but all were failed Thanks so much!

Answer: Instead of .cumcount() use sum: out = ( df.groupby(df["type"].ne(df["type"].shift()).cumsum(), as_index=False) .agg({"type": "first", "value": "sum"}) .rename(columns={"value": "grand"}) ) print(out)  Prints:  type grand 0 profit 21 1 loss -5 2 profit 65 3 loss -65