eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I have a DataFrame with column a. I would like to create two additional columns (b and c) based on column a. I could solve this problem doing the same thing twice: df = df.withColumn('b', when(df.a == 'something', 'x'))\ .withColumn('c', when(df.a == 'something', 'y'))  I would like to avoid doing the same thing twice, as the condition on which b and c are updated are the same, and also there are a lot of cases for column a. Is there a smarter solution to this problem? Could "withColumn" accept multiple columns perhaps?

Answer: A struct is best suited in such a case. See below example. spark.sparkContext.parallelize([('something',), ('foobar',)]).toDF(['a']). \ withColumn('b_c_struct', func.when(func.col('a') == 'something', func.struct(func.lit('x').alias('b'), func.lit('y').alias('c')) ) ). \ select('*', 'b_c_struct.*'). \ show() # +---------+----------+----+----+ # | a|b_c_struct| b| c| # +---------+----------+----+----+ # |something| {x, y}| x| y| # | foobar| null|null|null| # +---------+----------+----+----+  Just use a drop('b_c_struct') after the select to remove the struct column and keep the individual fields.
Question: VS Code automatically deletes unused imports and even more annoyingly it deletes imports that are used, but commented out. So for example, if I was to save this code: from pprint import pprint # pprint("foo")  It would remove the first line. So how can I turn this feature off, because it constantly forces me to rewrite the imports?

Answer: Check your settings.json (User and Workspace) for the following settings, delete this configuration or change it to false.  "editor.codeActionsOnSave": { "source.fixAll": true, },
Question: Problem statement I am creating a distribution plot of flood events per N year periods starting in 1870. I am using Pandas and Seaborn. I need help with...  specifying the date range of each bin when using sns.displot, and clearly representing my bin size specifications along the x axis.  To clarify this problem, here is the data that I am working with, what I have tried, and a description of the desired output. The Data The data I am using is available from the U.S. Weather service. import pandas as pd import bs4 import urllib.request link = "https://water.weather.gov/ahps2/crests.php?wfo=jan&gage=jacm6&crest_type=historic" webpage=str(urllib.request.urlopen(link).read()) soup = bs4.BeautifulSoup(webpage) tbl = soup.find('div', class_='water_information') vals = tbl.get_text().split(r'\n') tcdf = pd.Series(vals).str.extractall(r'\((?P<Rank>\d+)\)\s(?P<Stage>\d+.\d+)\sft\son\s(?P<Date>\d{2}\/\d{2}\/\d{4})')\ .reset_index(drop=True) tcdf['Stage'] = tcdf.Stage.astype(float) total_crests_events = len(tcdf) tcdf['Rank'] = tcdf.Rank.astype(int) tcdf['Date'] = pd.to_datetime(tcdf.Date)  What works I am able to plot the data with Seaborn's displot, and I can manipulate the number of bins with the bins command. The second image is closer to my desired output. However, I do not think that it's clear where the bins start and end. For example, the first two bins (reading left to right) clearly start before and end after 1880, but the precise years are not clear. import seaborn as sns # fig. 1: data distribution using default bin parameters sns.displot(data=tcdf,x="Date") # fig. 2: data distribution using 40 bins sns.displot(data=tcdf,x="Date",bins=40)    What fails I tried specifying date ranges using the bins input. The approach is loosely based on a previous SO thread. my_bins = pd.date_range(start='1870',end='2025',freq='5YS') sns.displot(data=tcdf,x="Date",bins=my_bins)  This attempt, however, produced a TypeError TypeError: Cannot cast array data from dtype('O') to dtype('float64') according to the rule 'safe'  This is a long question, so I imagine that some clarification might be necessary. Please do not hesitate to ask questions in the comments. Thanks in advance.

Answer: Seaborn internally converts its input data to numbers so that it can do math on them, and it uses matplotlib's "unit conversion" machinery to do that. So the easiest way to pass bins that will work is to use matplotlib's date converter: sns.displot(data=tcdf, x="Date", bins=mpl.dates.date2num(my_bins))
Question: I'm following this tutorial to install Odoo 15 on Mac, but I'm getting this error when running pip install -r requirements.txt:  Error compiling Cython file: ------------------------------------------------------------ ... cdef load_traceback cdef Waiter cdef wait cdef iwait cdef reraise cpdef GEVENT_CONFIG ^ ------------------------------------------------------------ src/gevent/_gevent_cgreenlet.pxd:181:6: Variables cannot be declared with 'cpdef'. Use 'cdef' instead.  I have found several documents addressing cython errors, but none addressing the specific exception I'm getting.

Answer: Odoo 15 currently seems to be incompatible with Python 3.10. You can get rid of that error by upgrading gevent/greenlet to newer versions in the requirements file. I've successfully tried gevent 21.12.0 and greenlet 1.1.3, BUT then you'll run into more trouble due to changes in the Python collections package. Here is a link to the issue regarding the gevent error for reference. Downgrade to Python 3.9.0 for the moment to continue with the installation.
Question: I'm trying to replace a value inside a variable that I use within an f-string. In this case it is a single quote. For example: var1 = "foo" var2 = "bar '" print(f"{var1}-{var2}")  Now, I want to get rid of the single quote within var2, but do it directly in the print statement. I've tried: print(f"{var1}-{var2.replace("'","")}")  which gives me: EOL while scanning string literal. I do not want to impose a third variable, so no var3 = var2.replace(",","") etc... I would rather not use a regex, but if there is no other way, please tell me how to do it. What is the best way to solve this?

Answer: When the contents contains both ' and ", you can use a triple quoted string: >>> print(f'''{var1}-{var2.replace("'","")}''') foo-bar
Question: How do you match the first occurance of start, starting from the end of the string? I have tried it with a negative lookahead but instead I get start\nfoo\nmoo\nstart\nfoo\ndoo as match import re pattern='(start[\s\S]*$)(?=$)' string='start\nfoo\nmoo\nstart\nfoo\ndoo' re.search(pattern, string)  expected match: start\nfoo\ndoo

Answer: You can use this code: string='start\nfoo\nmoo\nstart\nfoo\ndoo' print (re.findall(r'(?s).*(\bstart\b.*)', string)) ##> ['start\nfoo\ndoo']  RegEx Breakup:  (?s): Enable single line or DOTALL mode to make dot match line break as well .*: Match longest possible match including line breaks (\bstart\b.*): Match word start and everything after that till end in capture group #1. \b are necessary to avoid it matching restart or starting words.  PS: Since .* is greedy in nature before start it consume longest possible string before matching last start
Question: I have no problems printing without headless mode, however once I enable headless mode, it just refuses to print a PDF. I'm currently working on an app with a GUI, so I'd rather not have the Selenium webdriver visible to the end user if possible. For this project I'm using an older version of Selenium, 4.2.0. That coupled with Python 3.9. import os from os.path import exists import json from selenium.webdriver.chrome.service import Service from webdriver_manager.chrome import ChromeDriverManager from selenium.webdriver import Chrome, ChromeOptions # Paths dir_path = os.getcwd() download_path = os.path.join(dir_path, "letters") chrome_path = os.path.join(dir_path, "chromium\\app\\Chrome-bin\\chrome.exe") user_data_path = os.path.join(dir_path, "sessions") website = "https://www.google.com/" def main(): print_settings = { "recentDestinations": [{ "id": "Save as PDF", "origin": "local", "account": "", }], "selectedDestinationId": "Save as PDF", "version": 2, "isHeaderFooterEnabled": False, "isLandscapeEnabled": True } options = ChromeOptions() options.binary_location = chrome_path options.add_argument("--start-maximized") options.add_argument('--window-size=1920,1080') options.add_argument(f"user-data-dir={user_data_path}") options.add_argument("--headless") options.add_argument('--enable-print-browser') options.add_experimental_option("prefs", { "printing.print_preview_sticky_settings.appState": json.dumps(print_settings), "savefile.default_directory": download_path, # Change default directory for downloads "download.default_directory": download_path, # Change default directory for downloads "download.prompt_for_download": False, # To auto download the file "download.directory_upgrade": True, "profile.default_content_setting_values.automatic_downloads": 1, "safebrowsing.enabled": True }) options.add_argument("--kiosk-printing") driver = Chrome(service=Service(ChromeDriverManager().install()), options=options) driver.get(website) driver.execute_script("window.print();") if exists(os.path.join(user_data_path, "Google.pdf")): print("YAY!") else: print(":(") if __name__ == '__main__': main() 

Answer: For anyone else coming across this with a similar issue, I fixed it by using the print method described here: Selenium print PDF in A4 format Using my example from above, I replaced: driver.execute_script("window.print();")  with: pdf_data = driver.execute_cdp_cmd("Page.printToPDF", print_settings) with open('Google.pdf', 'wb') as file: file.write(base64.b64decode(pdf_data['data']))  And that worked just fine for me.
Question: I have a column that contains data like Dummy data: df = pd.DataFrame(["Lyreco A-Type small 2i", "Lyreco C-Type small 4i", "Lyreco N-Part medium", "Lyreco AKG MT 4i small", "Lyreco AKG/ N-Type medium 4i", "Lyreco C-Type medium 2i", "Lyreco C-Type/ SNU medium 2i", "Lyreco K-part small 4i", "Lyreco K-Part medium", "Lyreco SNU small 2i", "Lyreco C-Part large 2i", "Lyreco N-Type large 4i"])  I want to create an extra column that strips the data and gives you the required part of the string(see below) in each row. The extracted column should look like this Column_1 Column_2 Lyreco A-Type small 2i A-Type Lyreco C-Type small 4i C-Type Lyreco N-Part medium N-Part Lyreco STU MT 4i small STU MT Lyreco AKG/ N-Type medium 4i AKG/ N-Type Lyreco C-Type medium 2i C-Type Lyreco C-Type/ SNU medium 2i C-Type/ SNU Lyreco K-part small 4i K-part Lyreco K-Part medium K-Part Lyreco SNU small 2i SNU Lyreco C-Part large 2i C-Part Lyreco N-Type large 4i N-Type  How can I extract column 2 from the first column?

Answer: Looking at the example you posted, it's enough to split the column values and return "the middle" items. You can make a simple function to encapsulate the logic and apply it to the dataframe. from math import floor df = pd.DataFrame( {'Columns_1': ["Lyreco A-Type small 2i", "Lyreco C-Type small 4i", "Lyreco N-Part medium", "Lyreco AKG MT 4i small", "Lyreco AKG/ N-Type medium 4i", "Lyreco C-Type medium 2i", "Lyreco C-Type/ SNU medium 2i", "Lyreco K-part small 4i", "Lyreco K-Part medium", "Lyreco SNU small 2i", "Lyreco C-Part large 2i", "Lyreco N-Type large 4i" ] } ) def f(row): blocks = row['Columns_1'].split() mid_index = 1 if len(blocks) <= 4 else floor(len(blocks)/2) return ' '.join(blocks[1:mid_index+1]) df['Columns_2'] = df.apply(f, axis=1) print(df)  Output:  Columns_1 Columns_2 0 Lyreco A-Type small 2i A-Type 1 Lyreco C-Type small 4i C-Type 2 Lyreco N-Part medium N-Part 3 Lyreco AKG MT 4i small AKG MT 4 Lyreco AKG/ N-Type medium 4i AKG/ N-Type 5 Lyreco C-Type medium 2i C-Type 6 Lyreco C-Type/ SNU medium 2i C-Type/ SNU 7 Lyreco K-part small 4i K-part 8 Lyreco K-Part medium K-Part 9 Lyreco SNU small 2i SNU 10 Lyreco C-Part large 2i C-Part 11 Lyreco N-Type large 4i N-Type
Question: I have a map of the US and am plotting winning margin in Presidential elections in Plotly Express. I want a winning margin of 0 to be displayed as white and the scale to diverge into red/blue. However, the color_continuous_scale keyword argument takes a range from 0-1, and where a winning margin of 0 falls in this 0-1 range varies each election. Is there a workaround for this that would allow me to define 0 as white, and the extremes as red/blue? i have tried to convert the winning margin of 0 to a fraction between 0 and 1 but this changes with every election cycle. fig = px.choropleth(merged, locationmode='USA-states', locations='state_po', animation_frame='year', color='R_margin_x', color_continuous_scale=[(0, 'blue'), (merged['R_margin_x'].min()/(merged['R_margin_x'].min()-merged['R_margin_x'].max()), 'white'), (1, 'red')], scope='usa') fig.show()  dataframe is of the form party_simplified state_po year DEMOCRAT LIBERTARIAN OTHER REPUBLICAN D_pct_x L_pct_x O_pct_x R_pct_x R_margin_x D_pct_y L_pct_y O_pct_y R_pct_y R_margin_y R_lean 0 AK 2008 123594.0 1589.0 7173.0 193841.0 37.889374 0.487129 2.198978 59.424520 21.535146 52.761558 0.388418 1.488455 45.361569 -7.39999 28.935135 1 AL 2008 813479.0 NaN 19794.0 1266546.0 38.740434 NaN 0.942653 60.316913 21.576479 52.761558 0.388418 1.488455 45.361569 -7.39999 28.976468 2 AR 2008 422310.0 4776.0 21514.0 638017.0 38.864660 0.439529 1.979906 58.715904 19.851245 52.761558 0.388418 1.488455 45.361569 -7.39999 27.251234 3 AZ 2008 1034707.0 12555.0 16102.0 1230111.0 45.115251 0.547423 0.702079 53.635248 8.519997 52.761558 0.388418 1.488455 45.361569 -7.39999 15.919987 4 CA 2008 8274473.0 67582.0 208064.0 5011781.0 61.012638 0.498323 1.534180 36.954859 -24.057780 52.761558 0.388418 1.488455 45.361569 -7.39999 -16.657790 

Answer: In plotly you can pass the midpoint for continuous scales: color_continuous_scale=px.colors.sequential.RdBu, color_continuous_midpoint=0.0  The RdBu scale goes from red to white to blue. By passing color_continuous_midpoint=0.0 you specify that 0.0 is in the middle of the scale, i.e. white. The other colors will be determined accordingly. The same goes if you use a custom continuous color scale.
Question: Suppose I have a pandas DataFrame like this  name col1 col2 col3 0 AAA 1 0 2 1 BBB 2 1 2 2 CCC 0 0 2  I want (a) the names of any columns that contain a value of 2 anywhere in the column (i.e., col1, col3), and (b) the names of any columns that contain only values of 2 (i.e., col3). I understand how to use DataFrame.any() and DataFrame.all() to select rows in a DataFrame where a value appears in any or all columns, but I'm trying to find COLUMNS where a value appears in (a) any or (b) all rows.

Answer: You can do what you described with columns: df.columns[df.eq(2).any()] # Index(['col1', 'col3'], dtype='object') df.columns[df.eq(2).all()] # Index(['col3'], dtype='object')
Question: I have a local script that lists buckets: import boto3 s3_client = boto3.client('s3') for bucket in s3_client.list_buckets()["Buckets"]: print(bucket['Name'])  when I execute it locally, it does just that. Now if I execute this code as a lambda on AWS and set the log level to DEBUG, like so: import boto3 import logging logger = logging.getLogger() logger.setLevel(logging.DEBUG) s3_client = boto3.client('s3', endpoint_url="http://localhost:4566") def lambda_handler(event, context): s3_client.list_buckets() return { "statusCode": 200 }  I get detailed logs such as the headers of the HTTP request that is sent to S3. But If I add these lies to my local script nothing changes. I've tried: import boto3 import logging logger = logging.getLogger() logger.setLevel(logging.DEBUG) s3_client = boto3.client('s3') for bucket in s3_client.list_buckets()["Buckets"]: print(bucket['Name'])  and import boto3 import logging logging.getLogger('boto3').setLevel(logging.DEBUG) logging.getLogger('botocore').setLevel(logging.DEBUG) logging.getLogger('s3transfer').setLevel(logging.DEBUG) s3_client = boto3.client('s3') for bucket in s3_client.list_buckets()["Buckets"]: print(bucket['Name'])  as suggested here and in both cases I get no logs. How can I get my local script the show the logs for what boto3 does under the hood?

Answer: This seems to work for me: import boto3 import logging boto3.set_stream_logger('', logging.DEBUG) s3_client = boto3.client('s3') for bucket in s3_client.list_buckets()["Buckets"]: print(bucket['Name'])  Spews out a whole bunch of stuff. From this doc
Question: I want to use pathlib.glob() to find directories with a specific name pattern (*data) in the current working dir. I don't want to explicitly check via .isdir() or something else. Input data This is the relevant listing with three folders as the expected result and one file with the same pattern but that should be part of the result. ls -ld *data drwxr-xr-x 2 user user 4,0K 9. Sep 10:22 2021-02-11_68923_data/ drwxr-xr-x 2 user user 4,0K 9. Sep 10:22 2021-04-03_38923_data/ drwxr-xr-x 2 user user 4,0K 9. Sep 10:22 2022-01-03_38923_data/ -rw-r--r-- 1 user user 0 9. Sep 10:24 2011-12-43_3423_data  Expected result [ '2021-02-11_68923_data/', '2021-04-03_38923_data/', '2022-01-03_38923_data/' ]  Minimal working example from pathlib import Path cwd = Path.cwd() result = cwd.glob('*_data/') result = list(result)  That gives me the 3 folders but also the file. Also tried the variant cwd.glob('**/*_data/').

Answer: The trailing path separator certainly should be respected in pathlib.glob patterns. This is the expected behaviour in shells on all platforms, and is also how the glob module works:  If the pattern is followed by an os.sep or os.altsep then files will not match.  However, there is a bug in pathlib that was fixed in bpo-22276, and merged in Python-3.11.0rc1 (see what's new: pathlib). In the meantime, as a work-around you can use the glob module to get the behaviour you want: $ ls -ld *data drwxr-xr-x 2 user user 4096 Sep 9 22:45 2022-01-03_38923_data drwxr-xr-x 2 user user 4096 Sep 9 22:44 2021-04-03_38923_data drwxr-xr-x 2 user user 4096 Sep 9 22:44 2021-02-11_68923_data -rw-r--r-- 1 user user 0 Sep 9 22:45 2011-12-43_3423_data  >>> import glob >>> res = glob.glob('*_data') >>> print('\n'.join(res)) 2022-01-03_38923_data 2011-12-43_3423_data 2021-02-11_68923_data 2021-04-03_38923_data  >>> res = glob.glob('*_data/') >>> print('\n'.join(res)) 2022-01-03_38923_data/ 2021-02-11_68923_data/ 2021-04-03_38923_data/
Question: I'm currently studying pandas and I come from an R/dplyr/tidyverse background. Pandas has a not-so-intuitive API and how would I elegantly rewrite such operation from dplyr using pandas syntax? library("nycflights13") library("tidyverse") delays <- flights %>% group_by(dest) %>% summarize( count = n(), dist = mean(distance, na.rm = TRUE), delay = mean(arr_delay, na.rm = TRUE) ) %>% filter(count > 20, dest != "HNL") 

Answer:  pd.DataFrame.agg method doesn't allow much flexibility for changing columns' names in the method itself  That's not exactly true. You could actually rename the columns inside agg similar to in R although it is a better idea to not use count as a column name as it is also an attribute:  delays = ( flights .groupby('dest', as_index=False) .agg( count=('year', 'count'), dist=('distance', 'mean'), delay=('arr_delay', 'mean')) .query('count > 20 & dest != "HNL"') .reset_index(drop=True) )
Question: I'm using the code from this link. https://devskrol.com/2021/12/27/choropleth-maps-using-python/ Here's my actual code. import plotly.express as px from urllib.request import urlopen import json with urlopen('https://raw.githubusercontent.com/plotly/datasets/master/geojson-counties-fips.json') as response: counties = json.load(response) #import libraries import pandas as pd import plotly.express as px fig = px.choropleth(df_mover, geojson=counties, locations='my_zip', locationmode="USA-states", color='switcher_flag', range_color=(10000, 100000), scope="usa" ) fig.update_layout(margin={"r":0,"t":0,"l":0,"b":0}) fig.show()  I'm simply trying to pass in data from my dataframe named df_movers, which has two fields: my_zip and switcher_flag. When I run this in a Jupyter notebook, it just runs and runs; it never stops. I'm only trying to plot 25 records, so it's not like there's too much data here. Finally, my_zip is data type object. Any idea what could be wrong here?

Answer: Since you did not provide any user data, I tried your code with data including US zip codes from here. I think the issue is that you don't need to specify the location mode. I specified county_fips for the location and population for the color fill. import plotly.express as px from urllib.request import urlopen import json with urlopen('https://raw.githubusercontent.com/plotly/datasets/master/geojson-counties-fips.json') as response: counties = json.load(response) #import libraries import pandas as pd us_zip = pd.read_csv('data/uszips.csv', dtype={'county_fips': str}) fig = px.choropleth(us_zip, geojson=counties, locations='county_fips', #locationmode="USA-states", color='population', range_color=(1000, 10000), scope="usa" ) fig.update_layout(margin={"r":0,"t":0,"l":0,"b":0}) fig.show()
Question: I'm trying to turn this list: ['USA', 'Canada', 'Japan']  into this dictionary: {'USA': 0, 'Canada': 0, 'Japan': 0}  Can it be achieved with a simple loop? How would you go about it? Thanks!

Answer: Use dict.fromkeys: lst = ["USA", "Canada", "Japan"] out = dict.fromkeys(lst, 0) print(out)  Prints: {'USA': 0, 'Canada': 0, 'Japan': 0}
Question: Note: I know there is probably an answer for this on StackOverflow already, I just can't find it. I need to do this: >>> lst = [1, 2, 3, 4, 5, 6] >>> first_two = lst.magic_pop(2) >>> first_two [1, 2] >>> lst [3, 4, 5, 6]  Now magic_pop doesn't exist, I used it just to show an example of what I need. Is there a method like magic_pop that would help me to do everything in a pythonic way?

Answer: Do it in two steps. Use a slice to get the first two elements, then remove that slice from the list. first_list = lst[:2] del lst[:2]  If you want a one-liner, you can wrap it in a function. def splice(lst, start = 0, end = None): if end is None: end = len(lst) partial = lst[start:end] del lst[start:end] return partial first_list = splice(lst, end = 2)
Question: import matplotlib.pyplot as plt import pandas as pd l1 = [1,2,3,4] l2 = [2,4,6,8] fig = plt.figure() def func(): plt.pause(1) plt.plot(l1,l2) plt.draw() plt.pause(1) input("press any key to continue...") plt.close(fig) plt.pause(1) while True: func() plt.pause(1)  This is the modified one: import matplotlib.pyplot as plt import pandas as pd l1 = [1,2,3,4] l2 = [2,4,6,8] fig = plt.figure() a = 1 def func(num): input(f"the {num}th window is not opened yet") plt.pause(1) plt.plot(l1,l2) plt.draw() print(f"the {num}th window is opened") plt.pause(1) input("press any key to continue...") plt.close(fig) plt.pause(1) print(f"the {num}th window is closed") while True: func(a) plt.pause(1) a+=1  If I don't use while True loop, it stops running as I press any key and that was my intention. However, if I run this code with while True loop, the figure window doesn't close even though I press any key or x button in the upper left. I think it is due to while True. I have no idea how to solve this problem, keeping while True. Please help me!  modified: I could see an opened window when "The 2th window is not opened yet" input message came out. Probably, the window would be the one from the first time of the loop, since the second window wasn't opened at that time. Why was the first window still there? I used plt.close() to close the window. 

Answer: The issue is not while True: as much as how you create your figures. Let's step through your process conceptually:  fig = plt.figure() creates a figure and stores the handle in fig. Then you call func, which draws on the figure and eventually calls plt.pause(1). You loop around and call func again. This time, plt.plot(l1, l2) creates a new figure, since there are no open figures. func calls plt.close(fig). But the handle that's stored in fig is not the new figure that you opened, so of course your figure won't close.  To close the correct figure, open the figure inside func. Using the object oriented API gives you more control regarding what you open, close, plot, etc: import matplotlib.pyplot as plt import pandas as pd l1 = [1, 2, 3, 4] l2 = [2, 4, 6, 8] def func(): fig, ax = plt.subplots() ax.plot(l1, l2) plt.show(block=False) plt.pause(1) input("press any key to continue...") plt.close(fig) plt.pause(1) while True: func()  Alternatively, you can just replace plt.close(fig) with plt.close(), which is equivalent to plt.close(plt.gcf()), so you don't have to know the handle to your new figure.