eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I am using Spyder 5.3.1 and Python 3.10.4 within a virtual environment, on Windows 10. I know that with Python 3.10 came the match statement. However, whenever I use the match keyword inside a script, the following error appears: Code Analysis Invalid syntax (pyflakes E)  But I can run the script correctly without any problem. So what can be the issue here? Moreover, if I try directly in the IPython console, the match keyword is immediately recognized.

Answer: It's not so much an issue with Spyder as it is with the Pyflakes system it uses for linting code. match is a new (soft) keyword in Python 3.10. Pyflakes 2.4.0 only supports up to Python 3.8 currently. Pyflakes 2.5.0 is not out yet, but will cover up to Python 3.11, and should lint the new match keyword properly.
Question: I basically want to apply this formula: ((New/Old)^4 - 1) * 100. To a data frame I have and create a new column called "Annualized Growth Rate" I have been working off of the FRED GDP data set. I have a data set that looks something like this (not the real numbers)     Index GDP     0 100   1 101   2 103   3 107     I want to add a column on the right that applies the formula I mentioned above so that the annualized growth rate numbers would start appearing in the index = 1 position     Index GDP Annualized_Growth_Rate     0 100 NaN   1 101 4.060   2 103 8.159   3 107 16.462     How can I go about doing this. I was originally trying to do it using iterrows() Something like for index,row in iterrows(): df['Annualized Growth Rate'] = df['GDP'].loc[index] / df.['GDP'].loc[index - 1]...  but then index position -1 is out of range. I'm assuming there is an easy way to go about this that I just don't know. I also know you aren't really supposed to use something like iterrows.

Answer: You can use shift to access the previous row and vectorial operations for subtraction, division, power, multiplication: df['Annualized_Growth_Rate'] = df['GDP'].div(df['GDP'].shift()).pow(4).sub(1).mul(100)  Output:  Index GDP Annualized_Growth_Rate 0 0 100 NaN 1 1 101 4.060401 2 2 103 8.159184 3 3 107 16.462528
Question: def alarm_clock(day, vacation): if day == 0 or day == 6 and vacation != True: return "10.00" else: return "off" print(alarm_clock(0, True))  Why does this return "10.00"? In my mind it should return "off". Yes, day is equal to 0, but vacation is True, and the IF-statements first line states that it should only be executed if vacation is not True.

Answer: In Python and binds tighter than or. So your statement is equivalent to this: if day == 0 or (day == 6 and vacation != True):  To get the correct result you must parenthesize the precedence yourself: if (day == 0 or day == 6) and vacation != True:
Question:  I need to ignore timestamps and loop through rows this way.  import pandas as pd import numpy as np time = ['11:50', '12:50', '13:50'] data_1 = {'time': time, 'n1': [1, 5, 8], 'n2': [2, 6 ,7], 'n3': [3, 7 ,6], 'n4': [4, 8, 5], } df1 = pd.DataFrame(data = data_1) df1  I am trying to multiply:  row 1 * (10^0) row 2 * (10^1) row 3 * (10^2) ... row n * (10^(n-1))  Before:      time n1 n2 n3 n4     0 11:50 1 2 3 4   1 12:50 5 6 7 8   2 13:50 8 7 6 5     Expected result:      time n1 n2 n3 n4     0 11:50 1 2 3 4   1 12:50 50 60 70 80   2 13:50 800 700 600 500    

Answer: You can use mul on index axis: df1.iloc[:, 1:] = df1.iloc[:, 1:].mul(10**df1.index, axis=0) print(df1) # Output time n1 n2 n3 n4 0 11:50 1 2 3 4 1 12:50 50 60 70 80 2 13:50 800 700 600 500  You can replace df1.index by np.arange(len(df1)) if your index is not a RangeIndex.
Question: I have a 1d PyTorch tensor containing integers between 0 and n-1. Now I need to create a 2d PyTorch tensor with n-1 columns, where each row is a sequence from 0 to n-1 excluding the value in the first tensor. How can I achieve this efficiently? Ex: n = 3 a = torch.Tensor([0, 1, 2, 1, 2, 0]) # desired output b = [ [1, 2], [0, 2], [0, 1], [0, 2], [0, 1], [1, 2] ]  Typically, the a.numel() >> n. Detailed Explanation: The first element of a is 0, hence it has to map to the sequence [0, 1, 2] excluding 0, which is [1, 2]. Similarly, the second element of a is 1, hence it has to map to [0, 2] and so on. PS: I actually have an additional batch dimension, which I've excluded here for simplicity. Hence, I need the solution to be easily extendable to one additional dimension.

Answer: We can construct a tensor with the desired sequences and index with tensor a. import torch n = 3 a = torch.Tensor([0, 1, 2, 1, 2, 0]) # using torch.tensor is recommended def exclude_gather(a, n): sequences = torch.nonzero(torch.arange(n) != torch.arange(n)[:,None], as_tuple=True)[1].reshape(-1, n-1) return sequences[a.long()] exclude_gather(a, n)  Output tensor([[1, 2], [0, 2], [0, 1], [0, 2], [0, 1], [1, 2]])  We can add a batch dimension with functorch.vmap from functorch import vmap n = 4 b = torch.Tensor([[0, 1, 2, 1, 3, 0],[0, 3, 1, 0, 2, 1]]) vmap(exclude_gather, in_dims=(0, None))(b, n)  Output tensor([[[1, 2, 3], [0, 2, 3], [0, 1, 3], [0, 2, 3], [0, 1, 2], [1, 2, 3]], [[1, 2, 3], [0, 1, 2], [0, 2, 3], [1, 2, 3], [0, 1, 3], [0, 2, 3]]])
Question: In my python code I need to extract AWS credentials AWS_SECRET_ACCESS_KEY and AWS_ACCESS_KEY_ID which are stored in the plain text file as described here: https://docs.aws.amazon.com/sdkref/latest/guide/file-format.html I know the name of the file: AWS_SHARED_CREDENTIALS_FILE and the name of profile: AWS_PROFILE. My current approach is to read and parse this file in python by myself to get AWS_SECRET_ACCESS_KEY and AWS_ACCESS_KEY_ID. But I hope there is already standard way to get it using boto3 or some other library. Please suggest.

Answer: Would something like this work for you, or am I misunderstanding the question? Basically start a session for the appropriate profile (or the default, I guess), and then query those values from the credentials object:  session = boto3.Session(profile_name=<...your-profile...>) credentials = session.get_credentials() print("AWS_ACCESS_KEY_ID = {}".format(credentials.access_key)) print("AWS_SECRET_ACCESS_KEY = {}".format(credentials.secret_key)) print("AWS_SESSION_TOKEN = {}".format(credentials.token))
Question: I am developing a software as part of my work, as soon as I make a call to the API to fetch data the GUI freezes, at first I understood the problem and transferred the functions to threads, the problem is that once I used the join() function the app froze again. What I would like is to wait at the same point in the function until the threads end and continue from the same point in the function, is there any way to do this in Python? threads = [] def call_api(self, query, index, return_dict): thread = threading.Thread( target=self.get_data, args=(query, index, return_dict)) self.threads.append(thread) thread.start() def get_all_tickets(self, platform): if platform == 'All': self.call_api(query1, 0, return_dict) self.call_api(query2, 1, return_dict) for thread in self.threads: thread.join() # The app freezes here # Is there a way to wait at this point asynchronously until the processes are complete and continue from that point without the GUI freezing? 

Answer: One possible option would be to use QThreads finished signal it emits that you can connect to with a slot that contains the remaining logic from your get_all_tickets method. threads = [] def call_api(self, query, index, return_dict): thread = QThread() worker = Worker(query, index, return_dict) worker.moveToThread(thread) thread.started.connect(worker.run) worker.finished.connect(thread.terminate) thread.finished.connect(self.continue_getting_all_tickets) self.threads.append(thread) thread.start() def get_all_tickets(self, platform): if platform == 'All': self.call_api(query1, 0, return_dict) self.call_api(query2, 1, return_dict) def continue_getting_all_tickets(self): # this will be called once for each and every thread created # if you only want it to run once all the threads have completed # you could do something like this: if all([thread.isFinished() for thread in self.threads]): # ... do something  The worker class could look something like this. class Worker(QObject): finished = Signal() def __init__(self, query, index, return_dict): super().__init__() self.query = query self.index = index self.return_dict = return_dict def run(self): # this is where you would put `get_data` code # once it is finished it should emit the finished signal self.finished.emit()  Hopefully this will help you find the right direction.
Question: This is a weird issue I have ran into and I can't find any solution for this across the internet. I was using selenium in google colab to scrape a website and my code was working completely fine. I woke up the next day and ran the code again without changing a single line and don't know how/why my code starting giving me this error, AttributeError: 'WebDriver' object has no attribute 'find_element_by_link_text'. Same for find_element_by_class_name and id etc. I then rechecked a previously working script just to confirm and that gave me the same error too. I am confused about what happened suddenly and the scripts started giving me these errors. How do I solve this? What am I doing wrong here? !pip install selenium !apt-get update !apt install chromium-chromedriver from selenium import webdriver chrome_options = webdriver.ChromeOptions() chrome_options.add_argument('--headless') chrome_options.add_argument('--no-sandbox') chrome_options.add_argument('--disable-dev-shm-usage') driver = webdriver.Chrome('chromedriver',options=chrome_options) driver.get("https://petrowiki.spe.org/PetroWiki") driver.title #this line is returning the correct title value, code is able to access the url peh = driver.find_element_by_link_text('Pet. Eng. Handbook') peh.click() 

Answer: Selenium just removed that method in version 4.3.0. See the CHANGES: https://github.com/SeleniumHQ/selenium/blob/a4995e2c096239b42c373f26498a6c9bb4f2b3e7/py/CHANGES Selenium 4.3.0 * Deprecated find_element_by_* and find_elements_by_* are now removed (#10712) * Deprecated Opera support has been removed (#10630) * Fully upgraded from python 2x to 3.7 syntax and features (#10647) * Added a devtools version fallback mechanism to look for an older version when mismatch occurs (#10749) * Better support for co-operative multi inheritance by utilising super() throughout * Improved type hints throughout  You now need to use: driver.find_element("link text", "Pet. Eng. Handbook")  For improved reliability, you should consider using WebDriverWait in combination with element_to_be_clickable.
Question: i am currently trying to match specific sentences based on the words they contain and their order. I am doing this mostly with the lookahead assertion based on this structure: [^>.\]]*(?="The desired Words)[^<.\]]* So i am for example looking for sentences that talk about Vacation in the Maledives. To match the sentence: i´ll book a vacation to the Maledives. I could look for sentences that contain the word vacation and afterwards maledives. The expression [^>.\]]*(?=([Vv]cation.*[Mm]aledives))[^<.\]]* using .* causes problems because the word "Maledives" can also appear in later sentences (Example Wrong) My solutionw as to use the expression ([,'`´() \\]*\w+){0,X}\s* instead of .*, to indicate that "Maledives" has to follow "vacation" within the same sentences and with a maximum of X words between them, changing to this structure: [^>.\]]*(?=([Vv]cation([,'`´() \\]*\w+){0,X}\s*[Mm]aledives))[^<.\]]* (Example Correct) Unfortunately this expression is quite computationally intensive and leads to catastrophic backtracking if the range {0,X} is set to high. Do you have any other suggestions how to look for sentences containing specific words in order?

Answer: You can try something like this. (?:^|\.)\s*([^.]*[Vv]acation[^.]*[Mm]aledives[^.]*(?:\.|$))  See demo. https://regex101.com/r/97UvCH/1 There is no need for lookahead .It will slow things down.
Question: How do I create an argument parser (argparse.ArgumentParser) from a Pydantic model? I have a Pydantic model: from pydantic import BaseModel, Field class MyItem(BaseModel): name: str age: int color: str = Field(default="red", description="Color of the item")  And I want to create an instance of MyItem using command line: python myscript.py --name Jack --age 10 --color blue  This should yield to: item = MyItem(name="Jack", age=10, color="blue") ... # Process the item  I would not like to hard-code the command-line arguments and I would like to create the command-line arguments dynamically from the Pydantic model.

Answer: I found an answer myself. Just:  create an argument parser, turn the fields of the model as arguments of the parser, parse the command-line arguments, turn the arguments as dict and pass them to the model and process the instance of the model  import argparse from pydantic import BaseModel, Field class MyItem(BaseModel): name: str age: int color: str = Field(default="red", description="Color of the item") def add_model(parser, model): "Add Pydantic model to an ArgumentParser" fields = model.__fields__ for name, field in fields.items(): parser.add_argument( f"--{name}", dest=name, type=field.type_, default=field.default, help=field.field_info.description, ) # 1. Create and parse command line arguments parser = argparse.ArgumentParser() # 2. Turn the fields of the model as arguments of the parser add_model(parser, MyItem) # 3. Parse the command-line arguments args = parser.parse_args() # 4. Turn the arguments as dict and pass them to the model item = MyItem(**vars(args)) # 5. Do whatever print(repr(item)) ...  You may also add subparsers if you wish to add more functionality to the parser: https://docs.python.org/3/library/argparse.html#argparse.ArgumentParser.add_subparsers
Question: I want to overlay a transparent video over top of an image using ffmpeg and python I am able to do this successfully through terminal, but I cannot get ffmpeg commands to work in python. The following command produces the result that I want in terminal when I am in the directory with the files there. ffmpeg -i head1.png -i hdmiSpitting.mov -filter_complex "[0:v][1:v] overlay=0:0" -pix_fmt yuv420p -:a copy output3.mov  In python, my code is simple: import os import subprocess command = "ffmpeg -i head1.png -i hdmiSpitting.mov -filter_complex \"[0:v][1:v] overlay=0:0\" -pix_fmt yuv420p -c:a copy output3.mov" subprocess.call(command,shell=True)  The code runs, there is no indication of an error, but no output is produced. What am I missing here?

Answer: In Windows one line with spaces should work, but in Linux we have to pass the arguments as list.  We can build the command as a list:  command = ['ffmpeg', '-i', 'head1.png', '-i', 'hdmiSpitting.mov', '-filter_complex', '[0:v][1:v]overlay=0:0', '-pix_fmt', 'yuv420p', '-c:a', 'copy', 'output3.mov']   We may also use shlex.split:  import shlex command = shlex.split('ffmpeg -i head1.png -i hdmiSpitting.mov -filter_complex "[0:v][1:v] overlay=0:0" -pix_fmt yuv420p -c:a copy output3.mov')     Adding -y argument: If the output file output3.mov already exists, FFmpeg prints a message: File 'output3.mov' already exists. Overwrite? [y/N] And waits for the user to press y. In some development environments we can't see the message. Add -y for overwriting the output if already exists (without asking): command = shlex.split('ffmpeg -y -i head1.png -i hdmiSpitting.mov -filter_complex "[0:v][1:v] overlay=0:0" -pix_fmt yuv420p -c:a copy output3.mov')   Path issues: There are cases when ffmpeg executable is not in the execution path. Using full path may be necessary. Example for Windows (assuming ffmpeg.exe is in c:\FFmpeg\bin): command = shlex.split('c:\\FFmpeg\\bin\\ffmpeg.exe -y -i head1.png -i hdmiSpitting.mov -filter_complex "[0:v][1:v] overlay=0:0" -pix_fmt yuv420p -c:a copy output3.mov')  In Linux, the default path is /usr/bin/ffmpeg.  Using shell=True is not recommended and considered "unsafe". For details see Security Considerations. The default is False, so we may use subprocess.call(command). Note: subprocess.run supposes to replace subprocess.call. See this post for details.  Creating log file by adding -report argument: In some development environments we can't see FFmpeg messages, which are printed to the console (written to stderr). Adding -report argument creates a log file with name like ffmpeg-20220624-114156.log. The log file may tell us what went wrong when we can't see the console. Example: import subprocess import shlex subprocess.run(shlex.split('ffmpeg -y -i head1.png -i hdmiSpitting.mov -filter_complex "[0:v][1:v] overlay=0:0" -pix_fmt yuv420p -c:a copy output3.mov -report'))
Question: I apply idxmin() to get the index of the minimum absolute value of two columns: rep["OffsetFrom"] = rep[["OffsetDates", "OffsetDays"]].abs().idxmin(axis=1).dropna()   OffsetDates OffsetDays OffsetFrom 0 0.0 0.0 OffsetDates 1 1.0 1.0 OffsetDates 2 4.0 -3.0 OffsetDays 3 4.0 -3.0 OffsetDays 4 6.0 -1.0 OffsetDays ... ... ... ... 1165 0.0 0.0 OffsetDates 1166 0.0 0.0 OffsetDates 1167 0.0 0.0 OffsetDates 1168 0.0 0.0 OffsetDates 1169 0.0 0.0 OffsetDates 1170 rows × 3 columns  How do I get a column of the actual values after that? PS. This one wouldn't work, since I need to keep the original signs: rep[["OffsetFrom", "Offset"]] = rep[["OffsetDates", "OffsetDays"]].abs().agg(['idxmin','min'], axis=1)  Edit: Eventually, I've simply done it with apply: def get_abs_min_keep_sign(x): return min(x.min(), x.max(), key=abs) rep["Offset"] = rep[["OffsetDates", "OffsetDays"]].apply(get_abs_min_keep_sign, axis=1)  However, there must be a more elegant approach out there.

Answer: First idea is use lookup by column Offset: rep["Offset"] = rep[["OffsetDates", "OffsetDays"]].abs().idxmin(axis=1) idx, cols = pd.factorize(rep['Offset']) rep["OffsetVal"] = rep.reindex(cols, axis=1).to_numpy()[np.arange(len(rep)), idx]  Another numpy solution is use numpy.argmin for positions by absolute values, then is possible use indexing by array by columns names for Offset and for values is used indexing by rep[["OffsetDates", "OffsetDays"]] converted to 2d array: cols = ["OffsetDates", "OffsetDays"] pos = np.argmin(rep[cols].abs().to_numpy(), axis=1) rep["Offset2"] = np.array(cols)[pos] rep["OffsetVal2"] = rep[cols].to_numpy()[np.arange(len(rep.index)),pos] print (rep) OffsetDates OffsetDays Offset OffsetVal Offset2 OffsetVal2 0 0.0 0.0 OffsetDates 0.0 OffsetDates 0.0 1 1.0 1.0 OffsetDates 1.0 OffsetDates 1.0 2 4.0 -3.0 OffsetDays -3.0 OffsetDays -3.0 3 4.0 -3.0 OffsetDays -3.0 OffsetDays -3.0 4 6.0 -1.0 OffsetDays -1.0 OffsetDays -1.0  Your function should be simplify: rep["OffsetVal"] = rep[["OffsetDates","OffsetDays"]].apply(lambda x: min(x,key=abs),axis=1)
Question: How do you make an abstract property in python? import abc class MyClass(abc.ABC): @abc.abstractmethod @property def foo(self): pass  results in the error AttributeError: attribute '__isabstractmethod__' of 'property' objects is not writable

Answer: It turns out that order matters when it comes to python decorators. @abc.abstractmethod @property  is not the same as @property @abc.abstractmethod  The correct way to create an abstract property is: import abc class MyClass(abc.ABC): @property @abc.abstractmethod def foo(self): pass
Question: I have a dataset with 2 features with the name pos_x and pos_y and I need to scatter plot the clustered data done with DBScan. Here is what I have tried for it: dataset = pd.read_csv(r'/Users/file_name.csv') Data = dataset[["pos_x","pos_y"]].to_numpy() dbscan=DBSCAN() clusters =dbscan.fit(Data) p = sns.scatterplot(data=Data, x="pos_x", y="pos_y", hue=clusters.labels_, legend="full", palette="deep") sns.move_legend(p, "upper right", bbox_to_anchor=(1.17, 1.2), title='Clusters') plt.show()  however I get the following error for it. I appreciate if anyone can help me with it. Because as I know for the parameter x and y in scatter plot I should write the name of the features. ValueError: Could not interpret value `pos_x` for parameter `x` 

Answer: I think the error is caused by this part of the code: Data = dataset[["pos_x","pos_y"]].to_numpy()  When you convert the dataframe to numpy, seaborn cannot access the columns as it should. Try this: dataset = pd.read_csv(r'/Users/file_name.csv') Data = dataset[["pos_x","pos_y"]] dbscan = DBSCAN() clusters = dbscan.fit(Data.to_numpy()) p = sns.scatterplot(data=Data, x="pos_x", y="pos_y", hue=clusters.labels_, legend="full", palette="deep") sns.move_legend(p, "upper right", bbox_to_anchor=(1.17, 1.2), title='Clusters') plt.show()
Question: I have several images (PNG format) that I want to combine them into one image file in a grid structure (in such a way that I can set the No. of images shown in every row). Also, I want to add small empty space between images. For example, assume that there are 7 images. And I want to set the No. of images shown in every row as 3. The general structure of the combined image will be:  Please let me know if you know a good way to do that (preferably using PIL/Pillow or matplotlib libraries). Thanks.

Answer: You can pass to the combine_images function number of expected columns, space between images in pixels and the list of images: from PIL import Image def combine_images(columns, space, images): rows = len(images) // columns if len(images) % columns: rows += 1 width_max = max([Image.open(image).width for image in images]) height_max = max([Image.open(image).height for image in images]) background_width = width_max*columns + (space*columns)-space background_height = height_max*rows + (space*rows)-space background = Image.new('RGBA', (background_width, background_height), (255, 255, 255, 255)) x = 0 y = 0 for i, image in enumerate(images): img = Image.open(image) x_offset = int((width_max-img.width)/2) y_offset = int((height_max-img.height)/2) background.paste(img, (x+x_offset, y+y_offset)) x += width_max + space if (i+1) % columns == 0: y += height_max + space x = 0 background.save('image.png') combine_images(columns=3, space=20, images=['apple_PNG12507.png', 'banana_PNG838.png', 'blackberry_PNG45.png', 'cherry_PNG635.png', 'pear_PNG3466.png', 'plum_PNG8670.png', 'strawberry_PNG2595.png'])  Result for 7 images and 3 columns:  Result for 6 images and 2 columns: