eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: Could someone explain the difference between these two executions? Here is my generator function: def g(n): try: yield n print("first") except BaseException as e: print(f"exception {e}") raise e finally: print("second")  When I execute: >>> a = next(g(2)) exception second  Could someone explain why a = next(g(2)) raises an exception and what is the difference with the below execution? >>> x = g(2) >>> next(x) 2  My expectation was that when I execute a = next(g(2)) the function g(2) returns a generator and next() returns the first yield of the generator but apparently an exception is raised. Why is this happening?

Answer: In the first scenario, the generator is garbage collected after next(), triggering finally and raising an exception. In the second, the generator is kept by x, preventing this and allowing normal operation. Edit: GeneratorExit exception is thrown when a generator is closed, after that finally block will be exceuted. Also, from the docs:  If the generator is not resumed before it is finalized (by reaching a zero reference count or by being garbage collected), the generator-iterator’s close() method will be called, allowing any pending finally clauses to execute.
Question: I am trying to pull out some fields from a JSONn string into a dataframe. I can achieve this by put each field in a dataframe then join all the dataframes like below. But is there some easier way to do this? Because this is just an simplified example and I have a lot more fields to extract in my project.  from pyspark.sql import Row s = '{"job_id":"123","settings":{"task":[{"taskname":"task1"},{"taskname":"task2"}]}}' json_object = json.loads(s) # json_object job_id_l = [Row(job_id=json_object['job_id'])] job_id_df = spark.createDataFrame(job_id_l) # display(job_id_df) tasknames = [] for t in json_object['settings']["task"]: tasknames.append(Row(taskname=t["taskname"])) tasknames_df = spark.createDataFrame(tasknames) # display(tasknames_df) job_id_df.crossJoin(tasknames_df).display()  Result:  job_id taskname 123 task1 123 task2 

Answer: Update1: You don't even have to explicitly define the schema here and instead, you may simply use schema_of_json as follows: from pyspark.sql import SparkSession from pyspark.sql.functions import from_json, col, explode, schema_of_json, lit spark = SparkSession.builder.getOrCreate() s = '{"job_id":"123","settings":{"task":[{"taskname":"task1"},{"taskname":"task2"}]}}' schema = schema_of_json(lit(s)) result_df = ( spark.createDataFrame([s], "string") .select(from_json(col("value"), schema).alias("data")) .select("data.job_id", explode("data.settings.task.taskname").alias("taskname")) ) result_df.show() # +------+--------+ # |job_id|taskname| # +------+--------+ # | 123| task1| # | 123| task2| # +------+--------+  As you mentioned there are a lot more fields - this will take away some work from your hands.  OriginalAnswer: An easier way is to mirroring the schema with JSON string s and using from_json as follows: from pyspark.sql import SparkSession from pyspark.sql.functions import from_json, col, explode spark = SparkSession.builder.getOrCreate() s = '{"job_id":"123","settings":{"task":[{"taskname":"task1"},{"taskname":"task2"}]}}' schema = "struct<job_id:string, settings:struct<task:array<struct<taskname:string>>>>" result_df = ( spark.createDataFrame([s], "string") .select(from_json(col("value"), schema).alias("data")) .select("data.job_id", explode("data.settings.task.taskname").alias("taskname")) ) result_df.show() # +------+--------+ # |job_id|taskname| # +------+--------+ # | 123| task1| # | 123| task2| # +------+--------+
Question: class Solution: def isBalanced(self, root: Optional[TreeNode]) -> bool: balanced = [True] def node_height(root): if not root or not balanced[0]: return 0 left_height = node_height(root.left) right_height = node_height(root.right) if abs(left_height - right_height) > 1: balanced[0] = False return 0 return 1 + max(left_height, right_height) node_height(root) return balanced[0]  I understand why the code above works but when I change the value of variable 'balanced' to balanced = True instead of balanced = [True] in line 3 and change balanced[0] to balanced in line 6, 13 and 19, I get an error. class solution: def func(): a = 5 b = 7 c = True def nested_func(): return (a + b, c) return nested_func() print('sum is:', func())  So I tried the code above to test (maybe nested function cannot get variable that has boolean as a value) but was able to get the result of 'sum is: (12, True)' which shows that nested functions are able to get variables outside of them. Could someone explain this?

Answer: Nested functions can access variables from their parent functions. However, in your first case, not only you are accessing the value of the balanced variable, but you are also attempting to modify it. Whenever you create a variable in a nested function with the same name as a variable in the parent function, the nested function uses its own variable that is created inside the nested function. So in this code  def node_height(root): if not root or not balanced: return 0 left_height = node_height(root.left) right_height = node_height(root.right) if abs(left_height - right_height) > 1: balanced = False # You are creating the variable here, # but you are trying to access it above in the first if-else statement. # That is why you are getting the UnboundlocalError return 0 return 1 + max(left_height, right_height)  And about the first list approach, as you know list are mutable so you can update them inside the function, and it will update in all scope, so that is why it was working.  And in this example, As already Answered by Aemyl and mentioned by me above ("Nested functions can access variables from their parent functions."), In this code you are not updating the value instead you are just accessing the value of the parent's variable(s), so that is why you are not getting any error. class solution: def func(): a = 5 b = 7 c = True def nested_func(): return (a + b, c) return nested_func() print('sum is:', func())
Question: I need some help to translate a sql command to work on Flask. I am trying to filter the results of a query. This command below do what I need in mysql: select t.date, t.name from `capacitydata`.`allflash_dev` t inner join ( select name, max(date) as MaxDate from `capacitydata`.`allflash_dev` group by name) tm on t.name = tm.name and t.date = tm.MaxDate;  But I am trying without success to do the same on Flask. So far I got the query working, but showing all the lines # Creating Models class Block(db.Model): __tablename__ = "allflash_dev" index = db.Column(db.Date, nullable=False, unique=True, primary_key=True) date = db.Column(db.Date, nullable=False) name = db.Column(db.String(45), nullable=False) raw = db.Column(db.String(45), nullable=False) free = db.Column(db.String(45), nullable=False) frep = db.Column(db.String(45), nullable=False) util = db.Column(db.String(45), nullable=False) utip = db.Column(db.String(45), nullable=False) def create_db(): with app.app_context(): db.create_all() # Home route @app.route("/") def block(): details = Block.query.order_by(Block.date.desc()) return render_template("block.html", details=details)  Thanks in advance. I tried something like this but didn't work: details = db.select([Block.name, db.func.max(Block.date)]).group_by(Block.date, Block.name) 

Answer: You can use this. Use statement with your flask sqlalchemy session / query. from sqlalchemy import func, select from sqlalchemy.orm import aliased allflash_dev = aliased(Block) tm = ( select(allflash_dev.name, func.max(allflash_dev.date).label("MaxDate")) .group_by(allflash_dev.name) .subquery() ) statement = select(Block.date, Block.name).join( tm, (Block.name == tm.c.name) & (Block.date == tm.c.MaxDate) )  This generates the following sql. SELECT allflash_dev.date, allflash_dev.name FROM allflash_dev INNER JOIN ( SELECT allflash_dev_1.name AS name, max(allflash_dev_1.date) AS `MaxDate` FROM allflash_dev AS allflash_dev_1 GROUP BY allflash_dev_1.name ) AS anon_1 ON allflash_dev.name = anon_1.name AND allflash_dev.date = anon_1.`MaxDate`
Question: I have a pandas series.groupby objects, call it data. If I print out the elements, it looks like this: <pandas.core.groupby.generic.SeriesGroupBy object at ***> (1, 0 397.44 1 12.72 2 422.40 Name: value, dtype: float64) (2, 3 398.88 4 6.48 5 413.52 Name: value, dtype: float64) (3, 6 398.40 7 68.40 8 18.96 9 56.64 10 406.56 Name: value, dtype: float64) (4, 11 398.64 12 14.64 13 413.76 Name: value, dtype: float64) ...  I want to make an equivalent object, where the entries are the cumulative sum of each sublist in the series, minus the first entry of that list. So, for example, the first element would become: (1, 0 0 #(= 397.44 - 397.44) 1 12.72 #(= 397.44 + 12.72 - 397.44) 2 435.12 #(= 397.44 + 12.72 + 422.40 - 397.44)  I can get the cumulative sum easily enough using apply: cumulative_sums = data.apply(lambda x: x.cumsum()) but when I try to subtract the first element of the list in what I would think of as the intuitive way (lambda x: x.cumsum()-x[0]) , I get a KeyError. How can I achieve what I am trying to do?

Answer: Try: cumulative_sums = data.apply(lambda x: x.cumsum() - x.iat[0]) print(cumulative_sums)  Prints: a b 1 0 0.00 1 12.72 2 435.12 2 3 0.00 4 6.48 5 420.00 3 6 0.00 7 68.40 8 87.36 9 144.00 10 550.56 Name: value, dtype: float64
Question: I have this line of code that involves a matrix inversion: X = A @ B @ np.linalg.pinv(S)  A is an n by n matrix, B is an n by m matrix, and S is an m by m matrix. m is smaller than n but usually not orders of magnitude smaller. Usually m is about half of n. S is a symmetrical positive definite matrix. How do I make this line of code run faster in Python? I can do  X = np.linalg.solve(S.T, (A@B).T).T  But I am also curious if I can take advantage of the fact that S is symmetrical.

Answer: So your problem is XS = AB = C. As you've stated, this can be rewritten as S'X' = B'A' = C'. C is of size m x n, but this batched problem can be solved using scipy.linalg.solve. In this case, I recommend the scipy alternative (rather than numpy) because you have stated that S is symmetric, so you can pass assume_a="sym" argument so that scipy selects a solver that takes advantage of the matrix structure. So, your code will look like this: X = scipy.linalg.solve(S.T, (A@B).T, assume_a="sym").T
Question: Let's say I have a number of iterables: [[1, 2], [3, 4, 5, 6], [7, 8, 9], [10, 11, 12, 13, 14]]  How can I get only each element that is the first to appear at its index in any of the iterables? In this case: [1, 2, 5, 6, 14]  Visualized: [1, 2] [_, _, 5, 6] [_, _, _] [_, _, _, _, 14] 

Answer:  can it done in more functional style?  Sure, but I wouldn't. Davis Herring's approach is already lovely. Here's a "more functional" way, but more obscure to my eyes: from itertools import zip_longest SKIP = object() def chain_suffixes(*iters): return (next(a for a in s if a is not SKIP) for s in zip_longest(*iters, fillvalue=SKIP)) print(list(chain_suffixes( [1, 2], [3, 4, 5, 6], [7, 8, 9], [10, 11, 12, 13, 14])))  which prints  [1, 2, 5, 6, 14]  EDIT BTW, "more functional" may be in the eye of the beholder. To my eyes, various forms of generator comprehension are just as "functional" as other ways of spelling it. At an extreme, it's possible to rewrite the above without using comprehensions at all, or even typing "def". from itertools import zip_longest SKIP = object() not_SKIP = lambda x: x is not SKIP first_not_SKIP = lambda s: next(filter(not_SKIP, s)) chain_suffixes = lambda *iters: map( first_not_SKIP, zip_longest(*iters, fillvalue=SKIP))
Question: I would like to extract rows containing a particular string - the string can be a part of a larger, space-separated string (which I would want to count in), or can be a part of another (continuous) string (which I would NOT want to count in). The string can be either at start, middle or end of the string value. Example - say I would like to extract any row containing "HC": df = pd.DataFrame(columns=['test']) df['test'] = ['HC', 'CHC', 'HC RD', 'RD', 'MRD', 'CEA', 'CEA HC'] test 0 HC 1 CHC 2 HC RD 3 RD 4 MRD 5 CEA 6 CEA HC  Desired output  test 0 HC 2 HC RD 6 CEA HC 

Answer: You can use the str.contains method with the regex query \bHC\b >>> df[df['test'].str.contains(r'\bHC\b')] test 0 HC 2 HC RD 6 CEA HC  \b: Word boundary
Question: Let's say I have an array: a = [1,4,3,6,4]  I want to get an array where for every element I have the smallest value in a to the right (inclusive). That is, for a, I would want to create an array: [1,3,3,4,4]  Is there a quick, concise way to do this?

Answer: You question is actually ambiguous. Do you want to consider the following value or all following values? considering all following values compute a cumulated minimum on the reversed array with minimum.accumulate: a = np.array([1,4,3,6,4]) out = np.minimum.accumulate(a[::-1])[::-1]  Output: array([1, 3, 3, 4, 4])  With pure python and itertools.accumulate: from itertools import accumulate a = [1,4,3,6,4] out = list(accumulate(a[::-1], min))[::-1] # [1, 3, 3, 4, 4]  considering only the next value You could shift the values and use np.minimum: a = np.array([1,4,3,6,4]) out = np.minimum(a, np.r_[a[1:], a[-1]])  Output: array([1, 3, 3, 4, 4])  or compare the successive values and build a mask to modify a in place: a = np.array([1,4,3,6,4]) m = a[:-1] > a[1:] # or # m = np.diff(a) < 0 a[np.r_[m, False]] = a[1:][m]  Modified a: array([1, 3, 3, 4, 4])  difference a = np.array([1,4,3,6,5,6,4]) np.minimum.accumulate(a[::-1])[::-1] # array([1, 3, 3, 4, 4, 4, 4]) np.minimum(a, np.r_[a[1:], a[-1]]) # array([1, 3, 3, 5, 5, 4, 4])
Question: Question on calendar derivation logic in the Python module https://pypi.org/project/pandas-market-calendars/. Does this module depend on any third party API's to get the calendars or the calendars are derived based on rules within the code?

Answer: There is no communication with a third party API, everything is hardcoded. You can easily see this in the source code. There is a calendars folder with definitions of each calendar (see for example the file for ASX). The project's documentation also mentions that:  As of v2.0 this package provides a mirror of all the calendars from the exchange_calendars package, which itself is the now maintained fork of the original trading_calendars package. This adds over 50 calendars.  There is a list of available calendars here
Question: How to get the current domain name in Django template? Similar to {{domain}} for auth_views. I tried {{ domain }}, {{ site }}, {{ site_name }} according to below documentation. It didnt work. <p class="text-right">&copy; Copyright {% now 'Y' %} {{ site_name }}</p>  It can be either IP address 192.168.1.1:8000 or mydomain.com https://docs.djangoproject.com/en/5.0/ref/contrib/sites/  In the syndication framework, the templates for title and description automatically have access to a variable {{ site }}, which is the Site object representing the current site. Also, the hook for providing item URLs will use the domain from the current Site object if you don’t specify a fully-qualified domain.   In the authentication framework, django.contrib.auth.views.LoginView passes the current Site name to the template as {{ site_name }}. 

Answer: You can use {{ request.get_host }}
Question: I have the following pandas dataframe: d= {'Time': [0,1,2,0,1,2,2,3,4], 'Price': ['Auction', 'Auction','800','900','By Negotiation','700','250','250','Make Offer'],'Item': ['Picasso', 'Picasso', 'Picasso', 'DaVinci', 'DaVinci', 'DaVinci', 'Dali', 'Dali', 'Dali']} df = pd.DataFrame(data=d)  I would like to create a fourth column 'Listing-history' which would specify the following:  'first seen' if the listing is seen for the first time (this is not necessarily Time == 0) 'ongoing listing' if there is no change to the price field from one timepoint to the next 'Price->Auction' if the price field changes from a numeric value (which is actually encoded as a string in my dataframe) to the 'Auction' string, and vice versa if the price changes from a numeric value to the 'Auction' string. I would like the code to be agnostic to the exact price field string, for example: 'Price->By Negotiation' if the price field changes from a numeric value to 'By Negotiation'  I want to group by Item, and then apply the above logic. Finding whether a listing is 'first seen' is pretty straight forward using something like the following: df['Price_coerced_to_numeric'] = pd.to_numeric(df['Price'], errors='coerce') df['Price_diff'] = df.groupby(['Item'])['Price_coerced_to_numeric'].diff(1)  I suspect there is a way of using pandas apply and transform but I haven't been able to work it out. Any tips much appreciated.

Answer: You could use groupby.shift and numpy.select: # replace numbers by "Price" price = df['Price'].mask(pd.to_numeric(df['Price'], errors='coerce') .notna(), 'Price') # get previous price prev_price = price.groupby(df['Item']).shift() # identify first row per Item m1 = ~df['Item'].duplicated() # identify change in price m2 = price.ne(prev_price) # combine conditions df['Listing-history'] = np.select([m1, m2], ['first seen', prev_price+'->'+price], 'ongoing listing')  Output:  Time Price Item Listing-history 0 0 Auction Picasso first seen 1 1 Auction Picasso ongoing listing 2 2 800 Picasso Auction->Price 3 0 900 DaVinci first seen 4 1 By Negotiation DaVinci Price->By Negotiation 5 2 700 DaVinci By Negotiation->Price 6 2 250 Dali first seen 7 3 250 Dali ongoing listing 8 4 Make Offer Dali Price->Make Offer  If you really want to use groupby.transform you code refactor the code a bit: def history(col): price = col.mask(pd.to_numeric(col, errors='coerce').notna(), 'Price') prev_price = price.shift() return ((prev_price+'->'+price) .where(price.ne(prev_price), 'ongoing listing') .fillna('first seen') ) df['Listing-history'] = df.groupby('Item')['Price'].transform(history)  variant if you can have NaNs in the original column: def history(col): price = col.mask(pd.to_numeric(col, errors='coerce').notna(), 'Price') prev_price = price.shift() out = (prev_price+'->'+price).where(price.ne(prev_price), 'ongoing listing') out.iat[0] = 'first seen' return out df['Listing-history'] = df.groupby('Item')['Price'].transform(history)
Question: I want to make mean on groups of embeddings vectors. For examples: import polars as pl pl.DataFrame({ "id": [1,1 ,2,2], "values": [ [1,1,1], [3, 3, 3], [1,1,1], [2, 2, 2] ] })  shape: (4, 2) id values i64 list[i64] 1 [1, 1, 1] 1 [3, 3, 3] 2 [1, 1, 1] 2 [2, 2, 2]  Expected result. import numpy as np pl.DataFrame({ "id":[1,2], "values": np.array([ [[1,1,1], [3, 3, 3]], [[1,1,1], [2, 2, 2]] ]).mean(axis=1) })  shape: (2, 2) id values i64 list[f64] 1 [2.0, 2.0, 2.0] 2 [1.5, 1.5, 1.5] 

Answer:  int_ranges() to create ordinality for lists. explode() to explode lists. group_by() twice, first to calculate mean() and then to combine results into lists.  ( df .with_columns(i = pl.int_ranges(pl.col.values.list.len())) .explode('values', 'i') .group_by('id', 'i', maintain_order = True) .mean() .group_by('id', maintain_order = True) .agg('values') ) ┌─────┬─────────────────┐ │ id ┆ values │ │ --- ┆ --- │ │ i64 ┆ list[f64] │ ╞═════╪═════════════════╡ │ 1 ┆ [2.0, 2.0, 2.0] │ │ 2 ┆ [1.5, 1.5, 1.5] │ └─────┴─────────────────┘
Question: I installed pyperclip (a clipboard utility) on Windows VM and somehow some of the functions are not working. I am trying to use the waitForPaste() function as below: import pyperclip as pyc import os pyc.waitForPaste()  However, this is producing the following error: AttributeError Traceback (most recent call last) Cell In\[18\], line 1 \----\> 1 pyc.waitForPaste() AttributeError: module 'pyperclip' has no attribute 'waitForPaste  Strangely, if the following code works just fine! I was able to write this snippet: pyc.copy('The text to be copied to the clipboard.') pyc.paste()  Which produced: 'The text to be copied to the clipboard.'  I am not facing the same issue on MACOS, only in Windows.

Answer: A CTRL+F for "waitForPaste" in the pyperclip source code suggests that the function no longer exists despite still being in the documentation; the only hits it gets are in the documentation. You have a few options if you still need this functionality:  Find a workaround. This could be a similar function from another library or your own custom implementation of it. Find an old version. It seems like waitForPaste() was implemented at some time but later deleted; if you look back far enough in the history, you might be able to find a version that works. Ask for the feature back. You can try the pyperclip issues page, although there seems to be quite a bit of backlog there at the moment. Add the feature back yourself. pyperclip is open-source and publicly editable; if you write a working implementation or find the original implementation in the history, you can create a pull request to add it to the official source.  Of course, there might be a reason this feature was deleted; if that is the case, it's probably best to let sleeping dogs lie. Corroborating evidence (not crucial) The pyperclip docs contain the following code snippet and error message to demonstrate the timeout argument in waitForPaste(). >>> import pyperclip >>> pyperclip.waitForNewPaste(5) Traceback (most recent call last): File "<stdin>", line 1, in <module> File "c:\github\pyperclip\src\pyperclip\__init__.py", line 689, in waitForNewPaste raise PyperclipTimeoutException('waitForNewPaste() timed out after ' + str(timeout) + ' seconds.') pyperclip.PyperclipTimeoutException: waitForNewPaste() timed out after 5 seconds.  However, line 689 no longer exists in __init__.py; the last line is 658, which seems to imply that this function once existed but has since been deleted.
Question: I have been attempting to run 10 different looping tasks simultaneously with asyncio. All 10 tasks call "asyncio.create_task()" within their loops. I can't use "asyncio.run()" on all of them because this functions blocks the thread its called on until the task is done. So I thought that I could simply circumvent this by calling a function with "asyncio.run()" and then inside that function call my 10 looping functions through "asyncio.create_task()". The 10 looping functions are indeed called, but they themselves cannot use "asyncio.create_task()". Any suggestions on how to fix this issue? This is a based code I have written to demonstrate my issue: async def task2(): print("Task 2 was called") async def task1(): print("Task 1 was called") asyncio.create_task(task2()) async def main(): print("Main was called") asyncio.create_task(task1()) asyncio.run(main())  It prints: Main was called Task 1 was called  Any ideas or suggestions would be appreciated :D

Answer: You can create all the tasks and then asyncio.gather() them await asyncio.gather(*[task1(), task2()])
Question: import re x = re.compile(r"hello")  In the above code, x is determined to have type re.Pattern[str]. But why is re.Pattern generic, and then specialized to string? What does a re.Pattern[int] represent?

Answer: re.Pattern was made generic because you can also compile a bytes pattern that will operate only on bytes objects: p = re.compile(b'fo+ba?r') p.search(b'foobar') # fine p.search('foobar') # TypeError: cannot use a bytes pattern on a string-like object  At type-checking time, it is defined as generic over AnyStr: class Pattern(Generic[AnyStr]): ...  ...where AnyStr is a TypeVar with two constraints, str and bytes: AnyStr = TypeVar("AnyStr", str, bytes)  re.Pattern[int] is therefore meaningless and would cause a type-checking error.