eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I am reading a csv file and need to normalize the column names as part of a larger function chaining operation. I want to do everything with function chaining. When using the recommended name.map function for replacing chars in columns like: import polars as pl df = pl.DataFrame( {"A (%)": [1, 2, 3], "B": [4, 5, 6], "C (Euro)": ["abc", "def", "ghi"]} ).with_columns( pl.all().name.map( lambda c: c.replace(" ", "_") .replace("(%)", "pct") .replace("(Euro)", "euro") .lower() ) ) df.head()  I get shape: (3, 6) ┌───────┬─────┬──────────┬───────┬─────┬────────┐ │ A (%) ┆ B ┆ C (Euro) ┆ a_pct ┆ b ┆ c_euro │ │ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- │ │ i64 ┆ i64 ┆ str ┆ i64 ┆ i64 ┆ str │ ╞═══════╪═════|══════════╡═══════╡═════╡════════╡ │ 1 ┆ 4 ┆ "abc" ┆ 1 ┆ 4 ┆ "abc" │ │ 2 ┆ 5 ┆ "def" ┆ 2 ┆ 5 ┆ "def" │ │ 3 ┆ 6 ┆ "ghi" ┆ 3 ┆ 6 ┆"ghi" │ └───────┴─────┴──────────┴───────┴─────┴────────┘  instead of the expected shape: (3, 3) ┌───────┬─────┬────────┐ │ a_pct ┆ b ┆ c_euro │ │ --- ┆ --- ┆ --- │ │ i64 ┆ i64 ┆ str │ ╞═══════╪═════|════════╡ │ 1 ┆ 4 ┆ "abc" │ │ 2 ┆ 5 ┆ "def" │ │ 3 ┆ 6 ┆ "ghi" │ └───────┴─────┴────────┘  ? How can I replace specific chars in existing column names with function chaining without creating new columns?

Answer: You could simply replace DataFrame.with_columns() with DataFrame.select() method: df = pl.DataFrame( {"A (%)": [1, 2, 3], "B": [4, 5, 6], "C (Euro)": ["abc", "def", "ghi"]} ).select( pl.all().name.map( lambda c: c.replace(" ", "_") .replace("(%)", "pct") .replace("(Euro)", "euro") .lower() ) ) ┌───────┬─────┬────────┐ │ a_pct ┆ b ┆ c_euro │ │ --- ┆ --- ┆ --- │ │ i64 ┆ i64 ┆ str │ ╞═══════╪═════╪════════╡ │ 1 ┆ 4 ┆ abc │ │ 2 ┆ 5 ┆ def │ │ 3 ┆ 6 ┆ ghi │ └───────┴─────┴────────┘  IT would be important to say (as Dean MacGregor mentioned in the comments), that DataFrame.with_columns() always adds columns to the dataframe. The column names might be the same as the ones in the original dataframe, but in that case original columns will be replaced with the new ones. You can see it in the documentation:  Add columns to this DataFrame. Added columns will replace existing columns with the same name.  DataFrame.select(), on the other hand, selects existing columns of the dataframe. Additionally, if you just want to rename all the columns, it's probably more natural to use DataFrame.rename() instead: ... .rename( lambda c: c.replace(" ", "_") .replace("(%)", "pct") .replace("(Euro)", "euro") .lower() ) ┌───────┬─────┬────────┐ │ a_pct ┆ b ┆ c_euro │ │ --- ┆ --- ┆ --- │ │ i64 ┆ i64 ┆ str │ ╞═══════╪═════╪════════╡ │ 1 ┆ 4 ┆ abc │ │ 2 ┆ 5 ┆ def │ │ 3 ┆ 6 ┆ ghi │ └───────┴─────┴────────┘
Question: I have a dataset of investments that shows a value for the investment, a coupon rate, and an annual income. Some investments generate no income, and some investments have a coupon rate but the annual income is not computed. How do I update the income column when there is a valid coupon but no number in the income column? Here is some sample data: import pandas as pd import numbers data = {'value':[100, 150, 200, 250], 'coupon':[3,'-',4,5], 'income':[3,'-',8,'-']} df = pd.DataFrame(data)  Rows 0 and 2 are complete, and don't need updating. Row 1 has no value for income, and no value for coupon, so it also doesn't need updating.  Only the income value in the bottom row should be updated, to 12.5. Like in my dummy data set, the cells without meaningful numbers have dashes in them, they're not empty. I wrote a function to execute this algorithm, but I don't know how to apply it to the dataframe: def filler(value, coupon, income): if ~isinstance(income, numbers.Number) and isinstance(coupon, numbers.Number): income = (coupon * value)/100 return(income) else: pass 

Answer: You should not use a loop, not use - for missing values. Remove the -, use NaNs, and vectorize your code: df[['coupon', 'income']] = df[['coupon', 'income']].apply(pd.to_numeric, errors='coerce') df['income'] = df['income'].fillna(df['coupon'].mul(df['value']).div(100))  Output:  value coupon income 0 100 3.0 3.0 1 150 NaN NaN 2 200 4.0 8.0 3 250 5.0 12.5
Question: Updated: May I know how recursion works for this similar code? def fib( n ): global cnt cnt += 1 #global cnt is assigned and changed if n <= 2: return 1 return fib( n - 1 ) + fib( n - 2 ) cnt = 0 print(fib( 10 )) print("fib is called", cnt, "times") #109 times!  Old question: def power_of_two(n): global cnt cnt += 1 if n == 0: return 1 return 2 * power_of_two(n-1) cnt = 0 print(power_of_two(3)) print("power_of_two is called", cnt, "times")   Question: What is the value of cnt and how many times is the function power_of_two() called when the code above is executed? A. 8, 4 times B. 4, 3 times C. 6, 4 times D. 3, 3 times  I ran the code and it shows that cnt is 4, so should it be answer B? But for answer B, it says that the code is called 3 times but shouldn't it be 4? If it is 3, may I know why then? I thought of choosing answer A too since it makes more sense, but it asks for the value of cnt but not the value of (power_of_two(3)). Or did I misintepret the question/ did my teacher write the question wrongly? I know this is a basic programming question but thanks a lot for your help :)

Answer: it will be 4 times for n=3, Steps for your code. for n==3 your cnt ==1 return 2* #Recursion step 1 for n==2 it will be cnt ==2 return 2*2* #Recursion step 2 for n==1 it will be cnt==3 return 2*2*2* #Recursion step 3 and for last step is the stop recursion with if n==0 statement, so cnt==4 return 2*2*2*1 #Recursion step 4 result will be A) 8, 4 times
Question: Can I get the coordinates of markers to move the annotation to the top of the triangle? import matplotlib.pyplot as plt X = [1,2,3,4,5] Y = [1,1,1,1,1] labels = 'ABCDE' sizes = [1000, 1500, 2000, 2500, 3000] fig, ax = plt.subplots() ax.scatter(X, Y, s= sizes, marker = 10) for x, y, label, size in zip(X, Y, labels, sizes): print(x,y) ax.annotate(label, (x, y), fontsize=12) plt.show()  Which gives me: 

Answer: You need to (a) move the y coordinate by something proportional to the height of the marker; (b) root the text at bottom centre ("center" in American!). Note that the "size" of a marker is proportional to its AREA. So its height is proportional to sqrt(size). A certain amount of trial and error produced this. The height scaling probably depends on the type of marker. import math import matplotlib.pyplot as plt X = [1,2,3,4,5] Y = [1,1,1,1,1] labels = 'ABCDE' sizes = [1000, 1500, 2000, 2500, 3000] fig, ax = plt.subplots() ax.scatter(X, Y, s= sizes, marker = 10) for x, y, label, size in zip(X, Y, labels, sizes): print(x,y,size) ax.annotate(label, (x, y + math.sqrt( size ) / 3000 ), horizontalalignment='center', fontsize=12) plt.show()
Question: I've been exploring web scraping techniques using Python and RSS feed, but I'm not sure how to narrow down the search results to a particular year on Google News. Ideally, I'd like to retrieve headlines, publication dates, and possibly summaries for news articles from a specific year (such as 2020). With the code provided below, I can scrape the current data, but if I try to look for news from a specific year, it isn't available. Even when I use the Google articles search box, the filter only shows results from the previous year. However, when I scroll down, I can see articles from 2013 and 2017. Could someone provide me with a Python script or pointers on how to resolve this problem? Here's what I've attempted so far: import feedparser import pandas as pd from datetime import datetime class GoogleNewsFeedScraper: def __init__(self, query): self.query = query def scrape_google_news_feed(self): formatted_query = '%20'.join(self.query.split()) rss_url = f'https://news.google.com/rss/search?q={formatted_query}&hl=en-IN&gl=IN&ceid=IN%3Aen' feed = feedparser.parse(rss_url) titles = [] links = [] pubdates = [] if feed.entries: for entry in feed.entries: # Title title = entry.title titles.append(title) # URL link link = entry.link links.append(link) # Date pubdate = entry.published date_str = str(pubdate) date_obj = datetime.strptime(date_str, "%a, %d %b %Y %H:%M:%S %Z") formatted_date = date_obj.strftime("%Y-%m-%d") pubdates.append(formatted_date) else: print("Nothing Found!") data = {'URL link': links, 'Title': titles, 'Date': pubdates} return data def convert_data_to_csv(self): d1 = self.scrape_google_news_feed() df = pd.DataFrame(d1) csv_name = self.query + ".csv" csv_name_new = csv_name.replace(" ", "_") df.to_csv(csv_name_new, index=False) if __name__ == "__main__": query = 'forex rate news' scraper = GoogleNewsFeedScraper(query) scraper.convert_data_to_csv() 

Answer: You can use date filters in your rss_url. modify the query part in the below format Format: q=query+after:yyyy-mm-dd+before:yyyy-mm-dd Example: https://news.google.com/rss/search?q=forex%20rate%20news+after:2023-11-01+before:2023-12-01&hl=en-IN&gl=IN&ceid=IN:en The URL above returns articles related to forex rate news that were published between November 1st, 2023, and December 1st, 2023. Please refer to this article for more information.
Question: I have a pd dataframe: import pandas as pd column1 = [None,None,None,4,8,9,None,None,None,2,3,5,None] column2 = [None,None,None,None,5,1,None,None,6,3,3,None,None] column3 = [None,None,None,3,None,7,None,None,7,None,None,1,None] df = pd.DataFrame(np.column_stack([column1, column2,column3]),columns=['column1', 'column2', 'column3']) print(df) column1 column2 column3 0 None None None 1 None None None 2 None None None 3 4 None 3 4 8 5 None 5 9 1 7 6 None None None 7 None None None 8 None 6 7 9 2 3 None 10 3 3 None 11 5 None 1 12 None None None  I want to subset the rows between the values in column 3, and get rid of all empty rows. My desired outcomes are: print (df1) column1 column2 column3 0 4 None 3 1 8 5 None 2 9 1 7 print(df2) column1 column2 column3 0 None 6 7 1 2 3 None 2 3 3 None 3 5 None 1  I don't care about the actual values column3. Column 3 values are used to indicate "start" and "stop".

Answer: You can find the non-na value, then perform a cumulative sum, then mod 2 to get the "groups" of start and one-less-than stop positions. Shifting this by 1, adding to the original, and clipping to (0, 1) gets clumps of the start and stop points. To label the groups, you can take a diff of 1, then clip to (0, 1) again, and cum sum, then multiply those two together. g_small = (~df.column3.isna()).cumsum().mod(2) g = (g_small + g_small .shift(1, fill_value=0)).clip(0,1) groups = g.diff(1).fillna(0).clip(0,1).cumsum().astype(int) * g  You can then do a groupby operation on the data frame: dfs = {i: g for i, g in df.groupby(groups) if i > 0} dfs # returns: {1: column1 column2 column3 3 4 None 3 4 8 5 None 5 9 1 7, 2: column1 column2 column3 8 None 6 7 9 2 3 None 10 3 3 None 11 5 None 1}
Question: Problem description Suppose a following test class Foo: def __init__(self): self.value: int | None = None def set_value(self, value: int | None): self.value = value def test_foo(): foo = Foo() assert foo.value is None foo.set_value(1) assert isinstance(foo.value, int) assert foo.value == 1 # unreachable  The test:  First, checks that foo.value is something Then, sets the value using a method. Then it checks that the foo.value has changed.  When running the test with mypy version 1.9.0 (latest at the time of writing), and having warn_unreachable set to True, one gets: (venv) niko@niko-ubuntu-home:~/code/myproj$ python -m mypy tests/test_foo.py tests/test_foo.py:16: error: Statement is unreachable [unreachable] Found 1 error in 1 file (checked 1 source file)  What I have found  There is an open issue in the mypy GitHub: https://github.com/python/mypy/issues/11969 One comment said to use safe-assert, but after rewriting the test as  from safe_assert import safe_assert def test_foo(): foo = Foo() safe_assert(foo.value is None) foo.set_value(1) safe_assert(isinstance(foo.value, int)) assert foo.value == 1  the problem persists (safe-assert 0.4.0). This time, both mypy and VS Code Pylance think that foo.set_value(1) two lines above is not reachable. Question How can I say to mypy that the foo.value has changed to int and that it should continue checking also everything under the assert isinstance(foo.value, int) line?

Answer: You can explicitly control type narrowing with the TypeGuard special form (PEP 647). Although normally you would use TypeGuard to farther narrow a type than what has already been inferred, you can use it to 'narrow' to whatever type you choose, even if it is different or broader than the type checker has already inferred. In this case, we'll write a function _value_is_set which is annotated with a return type of TypeGuard[int] such that type checkers like mypy will infer type of int for values 'type guarded' under calls to this function (e.g., an assert of if expression). from typing import TypeGuard, Any # ... def _value_is_set(value: Any) -> TypeGuard[int]: if isinstance(value, int): return True return False def test_foo(): foo = Foo() assert foo.value is None foo.set_value(1) assert _value_is_set(foo.value) # the next line is redundant now, but can be kept without issue assert isinstnace(foo.value, int) assert foo.value == 1 # now reachable, according to mypy  Normally, mypy should treat assert isinstance(...) or if isinstance(...) in a similar way. But for whatever reason, it doesn't in this case. Using TypeGuard, we can coarse type checkers into doing the correct thing. With this change applied, mypy will not think this code is unreachable.
Question: I have been trying to plot a stacked plot with the same legend color and non-duplicate labels, without much of a success. import plotly.graph_objects as go from plotly.subplots import make_subplots # Sample data x = [1, 2, 3, 4, 5] y1 = [1, 2, 4, 8, 16] y2 = [1, 3, 6, 10, 15] y3 = [1, 4, 8, 12, 16] # Create subplot figure with two subplots fig = make_subplots(rows=1, cols=2, subplot_titles=('Subplot 1', 'Subplot 2')) # Add stacked area plot to subplot 1 fig.add_trace(go.Scatter(x=x, y=y1, mode='lines', stackgroup='one', name='A'), row=1, col=1) fig.add_trace(go.Scatter(x=x, y=y2, mode='lines', stackgroup='one', name='B'), row=1, col=1) # Add stacked area plot to subplot 2 fig.add_trace(go.Scatter(x=x, y=y1, mode='lines', stackgroup='two', name='A'), row=1, col=2) fig.add_trace(go.Scatter(x=x, y=y3, mode='lines', stackgroup='two', name='C'), row=1, col=2) # Update layout fig.update_layout(title_text='Stacked Area Plots on Subplots', showlegend=True) # Show figure fig.show()  The resulting plot is as follows:  Any help is appreciated.

Answer: You can set the fill color directly for each trace like so: from plotly.subplots import make_subplots import plotly.graph_objects as go # Sample data x = [1, 2, 3, 4, 5] y1 = [1, 2, 4, 8, 16] y2 = [1, 3, 6, 10, 15] y3 = [1, 4, 8, 12, 16] # Create subplot figure with two subplots fig = make_subplots(rows=1, cols=2, subplot_titles=("Subplot 1", "Subplot 2")) # Add stacked area plot to subplot 1 fig.add_trace(go.Scatter(x=x, y=y1, line={"color":"blue"}, fillcolor="blue", mode="lines", stackgroup="one", name="A", legendgroup="A"), row=1, col=1) fig.add_trace(go.Scatter(x=x, y=y2, line={"color":"red"}, fillcolor="red", mode="lines", stackgroup="one", name="B"), row=1, col=1) # Add stacked area plot to subplot 2 fig.add_trace(go.Scatter(x=x, y=y1, line={"color":"blue"}, fillcolor="blue", mode="lines", stackgroup="two", name="A", legendgroup="A", showlegend=False), row=1, col=2) fig.add_trace(go.Scatter(x=x, y=y3, line={"color":"red"}, fillcolor="red", mode="lines", stackgroup="two", name="C"), row=1, col=2) # Update layout fig.update_layout(title_text="Stacked Area Plots on Subplots", showlegend=True) # Show figure fig.show(renderer="browser") # or any other renderer  Or you can make a mapping like described [here].(https://community.plotly.com/t/automatically-pick-colors-when-using-add-trace/59075/2) I've used the argument legendgroup to group them and showlegend=False to hide one. The argument for line makes the color of the line match the fill.
Question: data={'category':['Medical','Medical','Research','Medical','Research'], 'countrycode':['US','CAN','US','CAN','US'], 'stateCode':['AK','AB','MO','NT','OK'], 'statecount':[600,100,200,760,90]} df=pd.DataFrame(data)  Given is my input dataframe. I want to produce a nested json output as follows : { 'Medical' : { 'US' : {'AK':600,'OK':90} 'CAN': {'AB':160, 'NT':760} }, 'Research' : { 'US' : {'MO':200,'OK':90} } }  I have been trying to play around with the following code, but cannot seem to get countrycode as second level. df.groupby('category').apply(lambda x: x.groupby('countrycode').apply(lambda y:y[['stateCode','statecount']]).to_json(orient='records'))  Any help or direction is appreciated, new to python.

Answer: One efficient option using groupby: out = {} for (k1,k2), g in df.groupby(['category', 'countrycode']): out.setdefault(k1, {})[k2] = g.set_index('stateCode')['statecount'].to_dict()  Alternatively, less efficient but maybe more flexible: {k1: {k2: g2.set_index('stateCode')['statecount'].to_dict() for k2, g2 in g.groupby('countrycode')} for k1, g in df.groupby('category')}  Output: {'Medical': {'CAN': {'AB': 100, 'NT': 760}, 'US': {'AK': 600}}, 'Research': {'US': {'MO': 200, 'OK': 90}}}
Question: if any([x % 2 for x in result]): print("good")  and if any(x % 2 for x in result): print("good")  I'm studying Python, but not sure what is difference between two expressions shown above. Does the first expression check each element in list? I try to code myself, to solve this problem, but I don't get it why those two expressions are different and what they do.

Answer: To see what's going on, let's run the following program: def test(i): print(i) if i == 5: return True return False if any([test(i) for i in range(10)]): print("Done") if any(test(i) for i in range(10)): print("Done")  0 1 2 3 4 5 6 7 8 9 Done 0 1 2 3 4 5 Done  The first version creates a list using list comprehension, and after the entire list is created, it is then checked for True/False values. You can see the entire list being created, because all numbers up to 10 are printed. However, in the second version, you'll see that the output only goes up to 5. This is because the second is a generator expression, which the any function than evaluates one-by-one. For example, as soon as i == 5, the any function returns true and stops iterating the generator. Let's replicate the any function in the same way it works in the second if-statement. def any(gen): for i in gen: if i: return True return False  This is pretty similar to how the builtin any function works. It looks at each element, checks if it is True, and returns before the loop is completed if a True value is found. The reason generator expressions are more efficient is because values are being created one-by-one, rather than all at once with the list comprehension.
Question: I have customer visit records with the channel they came from. I want to have one record per customer where I have the first channel they came from and the last channel they came from. Another logic I need to add is that if the first channel is "Direct", then do not take it and look at the next record. If that next record is also "Direct", then do not take it either and look at the next record. If you finished all records for the customer and they were all "Direct", then you can make first channel "Direct". I need the same logic for the last channel. Get the channel, if it is "Direct", then go to the previous one. If it is "Direct", then go through the previous record, until you finish all records. The input data looks like this:    Customer ID Date Channel     1 1/1/24 Email   1 1/2/24 Search   1 1/3/24 Direct   2 1/5/24 Direct   2 1/6/24 Paid   2 1/7/24 Email   3 1/8/24 Direct   3 1/9/24 Direct   3 1/10/24 Direct   3 1/11/24 Direct    And the output I need is this:    Customer ID First Channel Last Channel     1 Email Search   2 Paid Email   3 Direct Direct    How can I do that in my DataFrame? Probably I need to create two For loops, one to loop through all records, and inside it another loop to go through each customer record. It assigns first channel, then it checks if it is "Direct", then it goes to the next record, until the end of the loop.

Answer:  Convert Date to datetime so you can sort by Date (use format="%m/%d/%y" if your dates are in this format instead). Define condition_i and condition_ii to account for your new logic: you want to keep for each Customer ID: (i) all rows if Channel is Direct for all rows; or (ii) only rows where Channel is not Direct, if not all rows for Channel are Direct. Group on Customer ID then aggregate by first and last, respectively. Merge the dataframes resulting from the groupbys.  df["Date"] = pd.to_datetime(df["Date"], format="%d/%m/%y") df = df.sort_values(by="Date") condition_i = df.groupby("Customer ID")["Channel"].transform( lambda x: all(x == "Direct") ) condition_ii = ( df.groupby("Customer ID")["Channel"] .transform(lambda x: any(x != "Direct")) .loc[df["Channel"] != "Direct"] ) first = ( df[(condition_i) | (condition_ii)] .groupby("Customer ID", as_index=False) .first() .rename(columns={"Channel": "Fist Channel"}) ) last = ( df[(condition_i) | (condition_ii)] .groupby("Customer ID", as_index=False) .last() .rename(columns={"Channel": "Last Channel"}) ) out = pd.merge( first[["Customer ID", "Fist Channel"]], last[["Customer ID", "Last Channel"]] )   Customer ID Fist Channel Last Channel 0 1 Email Search 1 2 Paid Email 2 3 Direct Direct