eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: Looking for some guidance with regards to uploading files into AWS S3 bucket via a python script and an IAM role. I am able to upload files using BOTO3 and an aws_access_key_id & aws_secret_access_key for other scripts. However, I have now been given an IAM role to login to a certain account. I have no issue using AWS CLI to authenticate and query the S3 data so I do believe that my .aws/credential and .aws/config files are correct. However I am not sure how to use the ARN value within my python code. This is what I have put together so far, but get a variety of errors which all lead to denied access: session = boto3.Session(profile_name='randomName') session.client('sts').get_caller_identity() assumed_role_session = boto3.Session(profile_name='randomNameAccount') print(assumed_role_session.client('sts').get_caller_identity()) credentials = session.get_credentials() aws_access_key_id = credentials.access_key aws_secret_access_key = credentials.secret_key s3 = boto3.client('s3', aws_access_key_id=aws_access_key_id, aws_secret_access_key=aws_secret_access_key) bucket_name = 'bucketName'  This is a sample of what my credential and config files looks like as a referal. .aws/config file: [profile randomNameAccount] role_arn = arn:aws:iam::12345678910:role/roleName source_profile = randomName  aws/credentials file: [randomName] aws_access_key_id = 12345678910 aws_secret_access_key = 1234567-abcdefghijk  My question is help around the python code to be able to authenticate against AWS and navigate around a S3 bucket using an IAM role and then upload files when I call an upload function. Thank you in advance.

Answer: You should create an entry for the IAM Role in ~/.aws/credentials that refers to a set of IAM User credentials that have permission to assume the role: [my-user] aws_access_key_id = AKIAxxx aws_secret_access_key = xxx [my-role] source_profile = my-user role_arn = arn:aws:iam::123456789012:role/the-role  Add an entry to ~/.aws/config to provide a default region: [profile my-role] region = ap-southeast-2  Then you can assume the IAM Role with this code: import boto3 # Create a session by assuming the role in the named profile session = boto3.Session(profile_name='my-role') # Use the session to access resources via the role s3_client = session.client('s3') response = s3_client.list_objects(Bucket=...)
Question: I primarily use pathlib over os for paths, however there is one thing I have failed to get working on pathlib. If for example I require the latest created .csv within a directory I would use glob & os import glob import os target_dir = glob.glob("/home/foo/bar/baz/*.csv") latest_csv = max(target_dir, key=os.path.getctime)  Is there an alternative using pathlib only? (I am aware you can yield matching files with Path.glob(*.csv))

Answer: You can achieve it in just pathlib by doing the following: A Path object has a method called stat() where you can retrieve things like creation and modify date. from pathlib import Path files = Path("./").glob("*.py") latest_file = max([f for f in files], key=lambda item: item.stat().st_ctime) print(latest_file)
Question: Given a dataclass instance, I would like print() or str() to only list the non-default field values. This is useful when the dataclass has many fields and only a few are changed. @dataclasses.dataclass class X: a: int = 1 b: bool = False c: float = 2.0 x = X(b=True) print(x) # Desired output: X(b=True) 

Answer: The solution is to add a custom __str__() function: @dataclasses.dataclass class X: a: int = 1 b: bool = False c: float = 2.0 def __str__(self): """Returns a string containing only the non-default field values.""" s = ', '.join(f'{field.name}={getattr(self, field.name)!r}' for field in dataclasses.fields(self) if getattr(self, field.name) != field.default) return f'{type(self).__name__}({s})' x = X(b=True) print(x) # X(b=True) print(str(x)) # X(b=True) print(repr(x)) # X(a=1, b=True, c=2.0) print(f'{x}, {x!s}, {x!r}') # X(b=True), X(b=True), X(a=1, b=True, c=2.0)   This can also be achieved using a decorator: def terse_str(cls): # Decorator for class. def __str__(self): """Returns a string containing only the non-default field values.""" s = ', '.join(f'{field.name}={getattr(self, field.name)}' for field in dataclasses.fields(self) if getattr(self, field.name) != field.default) return f'{type(self).__name__}({s})' setattr(cls, '__str__', __str__) return cls @dataclasses.dataclass @terse_str class X: a: int = 1 b: bool = False c: float = 2.0
Question: I cannot find how to reorder columns in a polars dataframe in the polars DataFrame docs.

Answer: Turns out it is the same as pandas: df = df[['PRODUCT', 'PROGRAM', 'MFG_AREA', 'VERSION', 'RELEASE_DATE', 'FLOW_SUMMARY', 'TESTSUITE', 'MODULE', 'BASECLASS', 'SUBCLASS', 'Empty', 'Color', 'BINNING', 'BYPASS', 'Status', 'Legend']]
Question: I found a case when map() usage isn't equivalent to a list comprehension. It happens when next used as the first argument. For example: l1 = [1, 2] l2 = ['hello', 'world'] iterators = [iter(l1), iter(l2)] # list comprehension values1 = [next(it) for it in iterators] # values1 = [1, "hello"] values2 = [next(it) for it in iterators] # values2 = [2, "world"] values3 = [next(it) for it in iterators] # raise StopIteration  l1 = [1, 2] l2 = ['hello', 'world'] iterators = [iter(l1), iter(l2)] # map values1 = list(map(next, iterators)) # values1 = [1, "hello"] values2 = list(map(next, iterators)) # values2 = [2, "world"] values3 = list(map(next, iterators)) # values3 = [] # doesn't raise StopIteration  Any other exceptions occur as they should. Example: def divide_by_zero(value: int): return value // 0 l = [1, 2, 3] values = list(map(divide_by_zero, l)) # raises ZeroDivisionError as expected values = [divide_by_zero(value) for value in l] # raises ZeroDivisionError as expected, too  It seems very strange. It works the same with Python 3.9 and Python 3.11. It seems like map() works like this: def map(func, iterator): try: while True: item = next(iterator) yield func(item) except StopIteration: pass  but I expected it to work like this: def map(func, iterator): while True: try: item = next(iterator) except StopIteration: break yield func(item)  Is it a bug?

Answer: Try calling next on map: >>> >>> m = map(next, iterators) >>> next(m) 1 >>> next(m) 'hello' >>> next(m) Traceback (most recent call last): File "<stdin>", line 1, in <module> StopIteration  It's list that sees StopIteration and uses it to stop building the list from what map yields. The list comprehension, on the other hand, is building the list by iterating over iterators, not a particular iterator in that list. That is, next(it) is used to produce a value for the list, not to determine if we've reached the end of iterators.
Question: I have a string like aaabbbbcca  And I'd like to parse all possible uniform substrings from that. So my expected substrings for this string are ['a', 'aa', 'aaa', 'b', 'bb', 'bbb', 'bbbb', 'c', 'cc', 'a']  I tried the following import re print(re.findall(r"([a-z])(?=\1*)", "aaabbbbcca")) # Output: ['a', 'a', 'a', 'b', 'b', 'b', 'b', 'c', 'c', 'a']  Is it possible trough regular expressions? If yes, then how?

Answer: You can achieve what you need without a regex here: result = [] text = "aaabbbbcca" prev = '' for c in text: if c == prev: result.append(result[-1] + c) else: result.append(c) prev = c print(result) # => ['a', 'aa', 'aaa', 'b', 'bb', 'bbb', 'bbbb', 'c', 'cc', 'a']  See the Python demo. In short, you can iterate over the string and append new item to a result list when the new char is not equal to the previous char, otherwise, append a new item with the value equal to the previous item + the same char concatenated to the value. With regex, the best you can do is import re text = "aaabbbbcca" print( [x.group(1) for x in re.finditer(r'(?=((.)\2*))', text)] ) # => ['aaa', 'aa', 'a', 'bbbb', 'bbb', 'bb', 'b', 'cc', 'c', 'a']  See this Python demo. Here, (?=((.)\2*)) matches any location inside the string that is immediately preceded with any one char (other than line break chars if you do not use re.DOTALL option) that is followed with zero or more occurrences of the same char (capturing the char(s) into Group 1).
Question: Why SimpleNamespace code completion does not work in pycharm editor? from types import SimpleNamespace sn= SimpleNamespace(param_a = '1') sn. # pressing '.' dot I'm NOT offered param_a  This does work in pycharm python console, suggesting SimpleNamespace instance must be somehow 'computed' at runtime first. However if purpose of SimpleNamespace is to provide a namespace, to group a few parameters and access them via dot notation, then, while technically I can still type sn.param_a manually, without code completion is the whole thing stripped most of its usefulness and I'd likely switch to plain class where code completion does work in editor (class instantiated or not) Tried on different machines\pycharm versions so does not look like some quirk of my environment.

Answer:  Why SimpleNamespace code completion does not work in pycharm editor?  Because PyCharm doesn't have enough smarts to handle SimpleNamespaces specially (i.e. to know that the kwargs are just assigned into instance attributes).  This does work in pycharm python console, suggesting SimpleNamespace instance must be somehow 'computed' at runtime first  Yes, the console does something like vars() or dir() on the live object to introspect it. Static code analysis is an entirely different thing to inspecting objects at runtime. The instance isn't "computed" in any special way at runtime.  I'd likely switch to plain class where code completion does work in editor (class instantiated or not)  You might want to use dataclasses (or attrs) for brevity; both are well supported by PyCharm.
Question: I have came across a code that it is including descriptors. As I understand, __set_name__ is a method that is called when the class is created. Then, if the class is called twice I'd get two calls. In the following snippet I would expect to get the call in __set_name__ twice, but I am getting just one call. Why this behavior? class SharedAttribute: def __init__(self, initial_value=None): self.value = initial_value self._name = None def __get__(self, instance, owner): if instance is None: return self if self.value is None: raise AttributeError(f'{self._name} was never set') return self.value def __set__(self, instance, new_value): self.value = new_value def __set_name__(self, owner, name): print(f'{self} was named {name} by {owner}') self._name = name class GitFetcher: current_tag = SharedAttribute() current_branch = SharedAttribute() def __init__(self, tag, branch=None): self.current_tag = tag self.current_branch = branch @property def current_tag(self): if self._current_tag is None: raise AttributeError("tag was never set") return self._current_tag @current_tag.setter def current_tag(self, new_tag): self.__class__._current_tag = new_tag def pull(self): print(f"pulling from {self.current_tag}") return self.current_tag f1 = GitFetcher(0.1) f2 = GitFetcher(0.2) f1.current_tag = 0.3 f2.pull() f1.pull()  During the previous execution, __set_name__ is called with current_branch, but not called with current_tag. Why this distinction? The only call is this one: <__main__.SharedAttribute object at 0x047BACB0> was named current_branch by <class '__main__.GitFetcher'> 

Answer: TL;DR By the time __set_name__ methods are called, current_tag refers to an instance of property, not an instance of SharedAttribute.  __set_name__ is called after the class has been defined (so that the class can be passed as the owner argument), not immediately after the assignment is made. However, you changed the value of current_tag to be a property, so the name is no longer bound to a SharedAttribute instance once the class definition has completed.' From the documentation (emphasis mine):  Automatically called at the time the owning class owner is created.  The SharedAttribute instance is created while the body of the class statement is being executed. The class itself is not created until after the body is executed; the result of executing the body is a namespace which is passed as an argument to the metaclass that creates the class. During that process, the class attributes are scanned for values with a __set_name__ method, and only then is the method called. Here's a simpler example: class GitFetcher: current_branch = SharedAttribute() current_tag = SharedAttribute() current_tag = 3  By the time GitFetcher is defined, current_tag is no longer bound to a descriptor, so no attempt to call current_tag.__set_name__ is made. It's not clear if you want to somehow compose a property with a SharedAttribute, or if this is just an inadvertent re-use of the name current_tag.
Question: So this is my code basically: df = pd.read_csv('XBT_60.csv', index_col = 'date', parse_dates = True) df.index.freq = 'H'  I load a csv, set the index to the date column and want to set the frequency to 'H'. But this raises this error: ValueError: Inferred frequency None from passed values does not conform to passed frequency H  The format of the dates column is: 2017-01-01 00:00:00 I already tried loading the csv without setting the index column and used pd.to_datetime on the dates column before I set it as index, but still i am unable to set the frequency. How can I solve this? BTW: my aim is to use the seasonal_decompose() method from statsmodels, so I need the frequency there.

Answer: You can't set frequency if you have missing index values: >>> df val 2019-09-15 0 2019-09-16 1 2019-09-18 3 >>> df.index.freq = 'D' ... ValueError: Inferred frequency None from passed values does not conform to passed frequency D  To find missing index, use: >>> df = df.resample('D').first() val 2019-09-15 0.0 2019-09-16 1.0 2019-09-17 NaN 2019-09-18 3.0 >>> df.index.freq <Day>  To debug, find missing indexes: >>> pd.date_range(df.index.min(), df.index.max(), freq='D').difference(df.index) DatetimeIndex(['2019-09-17'], dtype='datetime64[ns]', freq=None)
Question: Code example: from typing import List class MyList(List[int]): def total(self) -> int: return sum(i for i in self) a = MyList([1,2,3]) print(f'{a.total()=:}')  When I run it, it works a.total()=6  But when I use pylint, I get the following error ... toy.py:5:30: E1133: Non-iterable value self is used in an iterating context (not-an-iterable) ...  There are other pylint errors, but they're understandable. For the not-an-iterable problem, I don't quite understand it, am I subclassing List[int], correctly? I'm using Python-3.8, pylint==2.6.0

Answer: I upgraded pylint to a more recent version. pylint --version pylint 2.9.6 astroid 2.6.6 Python 3.8.2 (default, Mar 15 2021, 10:18:42)  and the error is gone. Due to system constraints, I can't upgrade to the very latest version.
Question: I have a Pandas Dataframe df that details Names of players that play a game. The Dataframe has 2 columns of 'Date' they played a game and their name, sorted by Date.     Date Name     1993-03-28 Tom   1993-03-28 Joe   1993-03-29 Tom   1993-03-30 Joe     What I am trying to accomplish is to time-efficiently calculate the previous number of games each player has played before they play the upcoming game that day. For the example Dataframe above, calculating the players previous number of games would start at 0 and look like follows.     Date Name Previous Games     1993-03-28 Tom 0   1993-03-28 Joe 0   1993-03-29 Tom 1   1993-03-30 Joe 1     I have tried the following codes and although they have delivered the correct result, they took many days for my computer to run. Attempt 1: for i in range(0, len(df) ): df['Previous Games'][i] = len( df[ (df['Name'] == df['Name'][i]) & (df['Date'] < df['Date'][i]) ] )  Attempt 2: df['Previous Games'] = [ len( df[ (df['Name'] == df['Name'][i]) & (df['Date'] < df['Date'][i]) ] ) for i in range(0, len(df) ) ]  Although Attempt 2 was slightly quicker, it was still not time-efficient so I need help in finding a faster method.

Answer: Any time you write "for" and "pandas" anywhere close together you are probably doing something wrong. It seems to me you want the cumulative count: df["prev_games"] = df.sort_values('Date').groupby('Name').cumcount()
Question: According to the official guide https://docs.gunicorn.org/en/latest/settings.html#settings a worker will restart when it has handled max_requests requests. But when max_requests_jitter is set, a worker will restart when it has handled randint(0, max_requests_jitter) request, to stagger worker restarts to avoid all workers restarting at the same time. Is that means max_requests_jitter setting will override max_requests and make it be invalid？

Answer: From the docs -  The jitter causes the restart per worker to be randomized by randint(0, max_requests_jitter). This is intended to stagger worker restarts to avoid all workers restarting at the same time.  What I understand is that the jitter is a random addition to each worker and the term max_requests_jitter should be (though not necessary) smaller than max_requests. In other words, worker_1 will restart after max_requests + j1 requests, worker_2 will restart after max_requests + j2 requests etc. where the values of j1, j2, j3... are determined by the max_requests_jitter argument.
Question: I am trying to use the alive_progress alive_bar on PyCharm but it only appears in the console once the whole process has finished. Instead, I want it to display and progress as the for loop operates. Toy example: from alive_progress import alive_bar import time bar_l = 100 with alive_bar(bar_l) as bar: for i in range(bar_l): time.sleep(0.001) bar()  Has anyone else encountered this problem?

Answer: There is a option to force enable it and see alive-progress in PyCharm. It's "force_tty=True" with alive_bar(1000, force_tty=True) as bar: for i in range(1000): time.sleep(.01) bar()
Question: Is there an equivalent for Javascript's Object.assign(targetDict, srcDict) in python, which takes all the items from one dictionary into another, replacing as we go? (Cleaner than a for-in loop, anyhow) ------ Context --------- I use Object.assign in javascript to expand out settings-dictionary parameters in large functions, e.g.: function someFunctionWithLotsOfArguments(namedArg1, namedArg2, namedArg3, namedArg4){} // turns into function someFunctionWithLotsOfArguments(argDictionary){ namedArg1 = argDictionary["namedArg1"]; // etc }  The first form allows for default arguments while the second one does not, so I then use: function someFunctionWithLotsOfArguments(namedArg1="default1", namedArg2="default2", namedArg3="default3", namedArg4="default4"){} // turns into function someFunctionWithLotsOfArguments(argDictionary){ defaultArgs = { namedArg1: "default1", namedArg2: "default2", namedArg3: "default3", namedArg4: "default4" } Object.assign(defaultArgs, argDictionary); argDictionary = defaultArgs; namedArg1 = argDictionary["namedArg1"]; // etc }  I know, I know, this is a bit of an antipattern, and I should try and group my named arguments into structs to reduce my function parameter count, and if I have a large configurable function with many binary switches I should strip out the common functionality into smaller functions, etc. But if I did need a function like this, is there one available?

Answer: Python has a dict1.update(dict2) method which will do exactly that. :)
Question: I need to extract all matches from a string in a column and populate a second column. The matches will be delimited by a comma. df2 = pd.DataFrame([[1000, 'Jerry', 'string of text BR1001_BR1003_BR9009 more string','BR1003',''], [1001, '', 'BR1010_BR1011 random text', 'BR1010',''], ['', '', 'test to discardBR3009', 'BR2002',''], [1003, 'Perry','BR4009 pure gibberish','BR1001',''], [1004, 'Perry2','','BR1001','']], columns=['ID', 'Name', 'REGEX string', 'Member of','Status'])  Pattern representing the codes to be extracted. BR_pat = re.compile(r'(BR[0-9]{4})', re.IGNORECASE)  Hoped for output in column BR1001, BR1003, BR9009 BR1010,BR1011 BR3009 BR4009  My attempt: df2['REGEX string'].str.extractall(BR_pat).unstack().fillna('').apply(lambda x: ", ".join(x))  Output:  match 0 0 BR1001, BR1010, BR3009, BR4009 1 BR1003, BR1011, , 2 BR9009, , ,  There are extra commas and rows missing. What did I do wrong?

Answer: You need to use >>> df2['REGEX string'].str.findall(r'BR\d{4}').str.join(", ") 0 BR1001, BR1003, BR9009 1 BR1010, BR1011 2 BR3009 3 BR4009 4 Name: REGEX string, dtype: object  With Series.str.findall, you extract all occurrences of the pattern inside a string value, it returns a "Series/Index of lists of strings". To concatenate them into a single string, the Series.str.join() is used.
Question: Is the time complexity of nested for, while, and if statements the same? Suppose a is given as an array of length n. for _ in range(len(a)): for _ in range(len(a)): do_something  The for statement above will be O(n²). i = 0 while i < len(a) * len(a): do_something i += 1  At first glance, the above loop can be thought of as O(n), but in the end I think that it is also O(n²). Am I right?

Answer:  Am I right?  Yes! The double loop: for _ in range(len(a)): for _ in range(len(a)): do_something  has a time complexity of O(n) * O(n) = O(n²) because each loop runs until n. The single loop: i = 0 while i < len(a) * len(a): do_something i += 1  has a time complexity of O(n * n) = O(n²), because the loop runs until i = n * n = n².
Question: I have a dataframe like this import pandas as pd df = pd.DataFrame( { 'C1': list('aabbab'), 'C2': list('abbbaa'), 'value': range(11, 17) } ) C1 C2 value 0 a a 11 1 a b 12 2 b b 13 3 b b 14 4 a a 15 5 b a 16  and I would like to generate a dictionary like this: {'C1': {'a': {1: 11, 2: 12, 3: 15}, 'b': {1: 13, 2: 14, 3: 16}}, 'C2': {'a': {1: 11, 2: 15, 3: 16}, 'b': {1: 12, 2: 13, 3: 14}}}  Logic is as follows: In df I go to the column C1 and the first a I find in the column corresponds to value 11, the second one to value 12 and the third one to 15. The position of the a and the corresponding value should be stored in the dictionary for the keys C1 and a. I could do something like this df_ss = df.loc[df['C1'] == 'a', 'value'] d = {ind: val for ind, val in enumerate(df_ss.values, 1)}  which yields for d: {1: 11, 2: 12, 3: 15}  which is indeed the desired output. I could then put this into a loop and generate all required dictionaries. Does anyone sees something more efficient than this?

Answer: You could use a groupby and a nested dict comprehension: import pandas as pd df = pd.DataFrame( { 'C1': list('aabbab'), 'C2': list('abbbaa'), 'value': range(11, 17) } ) d = { c: {k: dict(enumerate(g["value"], 1)) for k, g in df.groupby(c)} for c in ["C1", "C2"] }  Which outputs: {'C1': {'a': {1: 11, 2: 12, 3: 15}, 'b': {1: 13, 2: 14, 3: 16}}, 'C2': {'a': {1: 11, 2: 15, 3: 16}, 'b': {1: 12, 2: 13, 3: 14}}}
Question: As the title says, how to find and replace text inside a single jupyter cell when using the jupyter extension in Visual Studio Code? I am familiar with ctr+h but that will replace all the occurrences in the entire jupyter notebook file. This is a really important feature for me, as I am using it a lot in jupyter on the browser.

Answer: You can select the first occurrence and then use Ctrl+D. It will select the next occurence in the cell. Repeat that until you go back to the first ocurrence and then type the new value. It will replace all the values your circled on. In case you have changed that keyboard shortcut or if it is different you can find the correct one: