eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I have created a pandas dataframe as follows: import pandas as pd import numpy as np ds = { 'trend' : [1,1,1,1,2,2,3,3,3,3,3,3,4,4,4,4,4], 'price' : [23,43,56,21,43,55,54,32,9,12,11,12,23,3,2,1,1]} df = pd.DataFrame(data=ds)  The dataframe looks as follows: display(df)  trend price 0 1 23 1 1 43 2 1 56 3 1 21 4 2 43 5 2 55 6 3 54 7 3 32 8 3 9 9 3 12 10 3 11 11 3 12 12 4 23 13 4 3 14 4 2 15 4 1 16 4 1  I have saved the dataframe to a .csv file called df.csv: df.to_csv("df.csv", index = False)  I need to create a new field called ema2 which:  iterates through each and every record of the dataframe  calculates the Exponential Moving Average (EMA) by considering the price observed at each iteration and the prices (EMA length is 2 in this example) observed in the previous trends. For example:  I iterate at record 0 and the EMA is NaN (missing).  I iterate at record 1 and the EMA is still NaN (missing)  I Iterate at record 12 and the EMA is 24.20 (it considers price at record 3, price at record 5 and price at record 12  I Iterate at record 13 and the EMA is 13.53 (it considers price at record 3, price at record 5 and price at record 13  I Iterate at record 15 and the EMA is 12.46 (it considers price at record 3, price at record 5 and price at record 15 and so on .....   I have written the following code: time_window = 2 ema= [] for i in range(len(df)): ds = pd.read_csv("df.csv", nrows=i+1) d = ds.groupby(['trend'], as_index=False).agg( {'price':'last'}) d['ema2'] = d['price'].ewm(com=time_window - 1, min_periods=time_window).mean() ema.append(d['ema2'].iloc[-1]) df['ema2'] = ema  Which produces the correct dataframe: print(df) trend price ema2 0 1 23 NaN 1 1 43 NaN 2 1 56 NaN 3 1 21 NaN 4 2 43 35.666667 5 2 55 43.666667 6 3 54 49.571429 7 3 32 37.000000 8 3 9 23.857143 9 3 12 25.571429 10 3 11 25.000000 11 3 12 25.571429 12 4 23 24.200000 13 4 3 13.533333 14 4 2 13.000000 15 4 1 12.466667 16 4 1 12.466667  The problem is that when the dataframe has millions of records: it takes a very long time to run. Does anyone know how to get the same results in a quick, efficient way, please?

Answer: I slightly changed the example that you asked about RSI. I added -1 in the first prev, the cycle of filling by slices, price and in setting the values ​​by slice of the data frame. You can also try numba, cython, but most likely the code will need to be rewritten(not all functions in them are available from numpy, I don't know about pandas). trends = df["trend"].unique() arr = df['price'].values range_group = np.stack( [df[df["trend"] == trend].index.values.take([0, -1]) for trend in trends] ) price = np.full((len(df), trends.size), np.nan) prev = arr[range_group[:time_window-1, 1]] for i in range(time_window-1, len(trends)): stop = range_group[i, 1] + 1 price[range_group[i, 0]:stop, -1] = arr[range_group[i, 0]:stop] price[range_group[i, 0]:stop, -(prev.size+1):-1] = prev prev = price[range_group[i, 1], -(prev.size+1):] price = price[range_group[time_window-1, 0]:] val = (pd.DataFrame(price).T.ewm(com=time_window - 1, min_periods=time_window).mean().iloc[-1].values) df.loc[range_group[time_window-1, 0]:, 'ema'] = val
Question: Using django: here is some values and the query:  max_played, avg_days_last_played = get_next_song_priority_values() # Calculate time since played using raw SQL time_since_played_expr = RawSQL("(julianday('now') - julianday(main_song.played_at))", []) query = Song.objects # Annotate priority songs_with_priority = query.annotate( time_since_played=ExpressionWrapper(time_since_played_expr, output_field=FloatField()), priority=( F('rating') - (F('count_played') / Value(max_played)) + (F('time_since_played') / Value(avg_days_last_played)) ), ).order_by('-priority')  my logging:  logger.info(f'Next Song: {next_song}') calculated_priority = ( next_song.rating - (next_song.count_played / max_played) + (next_song.time_since_played / avg_days_last_played) ) logger.info(f'Next Song: priority {next_song.priority:.2f} vs calc {calculated_priority:.2f}') logger.info(f'Next Song: rating {next_song.rating:.2f}') playd = next_song.count_played / max_played logger.info(f'Next Song: played {playd:.2f} ({next_song.count_played} / {max_played})') tspd = next_song.time_since_played / avg_days_last_played logger.info( f'Next Song: days {tspd:.2f} ({next_song.time_since_played} / {avg_days_last_played})' )  and I get: INFO Next Song: <Song-1489 End of the World Cold> INFO Next Song: priority 2.73 vs calc 2.56 INFO Next Song: rating 0.50 INFO Next Song: played 0.17 (1 / 6) INFO Next Song: days 2.23 (4.043296354357153 / 1.8125720233656466)  So my calculated value is lower. All the values are there: the rating of 0.5 is solid, the play counts if 1 vs 6 is solid, the time since is used from the result next_song.time_since_played. I'm using the same values sqlite should be using, but my calc is different.

Answer: In SQLite, integer division produces an integer value, so 1 / 6 = 0: sqlite> select 1 / 6; 0  You can multiply one of the value by 1.0 to convert it to a float value and then this should work: sqlite> select 1 / 6.0; 0.166666666666667   priority=( F('rating') - (F('count_played') / Value(max_played * 1.0)) # <- this line + (F('time_since_played') / Value(avg_days_last_played)) )  or explicitly convert max_played into a float: (F('count_played') / Value(float(max_played)))
Question: Say I have a number num and I want to count the number of positive integers in the range [1, n] which are lexicographically smaller than num and n is some arbitrary large integer. A number x is lexicographically smaller than a number y if the converted string str(x) is lexicographically smaller than the converted string str(y). I want to do this efficiently since n could be large (eg. 10^9). My idea for this is using digit dynamic programming. Essentially what I'm thinking is that every number in the range [1,n] can be represented as a string of len(str(n)) slots. At each slot, the upper bound for this is either the digit at the last position of num (this is for the case where we pick trailing zeros) or the digit at the last position of n. This is because if the previous digit is already smaller than the corresponding digit in num then we are free to pick any digit up to the corresponding digit in n. Below is my code in Python that attempts to do this from functools import cache def count(num, n): num = str(num) n = str(n) max_length = len(n) @cache def dp(indx, compare_indx, tight, has_number): if indx == max_length: return int(has_number) ans = 0 upper_bound = int(num[compare_indx]) if tight else int(n[indx]) for digit in range(upper_bound + 1): if digit == 0 and not has_number: ans += dp(indx + 1, compare_indx, tight and (digit == upper_bound), False) else: ans += dp(indx + 1, min(compare_indx + 1, len(num) - 1), tight and (digit == upper_bound), True) return ans return dp(0, 0, True, False)  However, count(7, 13) outputs 35 which is not correct since the lexicographical order of [1, 13] is [1, 10, 11, 12, 13, 2, 3, 4, 5, 6, 7, 8, 9] so count(7, 13) should be 10. Can anyone help me out here?

Answer: I couldn't follow the logic in your explanation, but this shouldn't need dynamic programming. In essence you want to do a separate count for each possible width of an integer. For instance, when calling count(7, 13), you'd want to count:  integers with one digit: [1, 6] = 6 integers integers with two digits: [10, 13] = 4 integers  The outcome is the sum: 6 + 4 = 10 Take count(86, 130) as another example, where the first argument has more than one digit:  integers with one digit: [1, 8] = 8 integers (note that 8 is included) integers with two digits: [10, 85] = 76 integers (note that 86 is excluded) integers with three digits: [100, 130] = 31 integers  Total is: 115 So some care has to be taken at the high-end of the ranges: when it is a proper prefix of the first argument, it should be included, if not, it should be excluded. And of course, for that last group (with the greatest number of digits) you should take care not to exceed the value of the second argument. Here is how you could code that logic: def count(num, n): strnum = str(num) lennum = len(strnum) max_length = len(str(n)) strnum += "0" * (max_length - lennum) # pad with zeroes at the right count = 0 low = 1 for width in range(1, max_length + 1): high = int(strnum[:width]) addone = width < lennum and n >= high count += min(high, n + 1) - low + addone low *= 10 return count
Question: I have 2 lists, one that contains both numbers and strings and one only numbers: list1 = [1, 2, 'A', 'B', 3, '4'] list2 = [1, 2, 3, 4, 5, 6]  My goal is to print from list2 only the numbers that have another number (both as number or string) in the same index in list1. Expected output: [1,2,5,6]  I have tried the following code: lenght1 = len(list1) for i in range(lenght1): if (list1[i].isdigit()): print(list2[i])  But I receive the following error: AttributeError: 'int' object has no attribute 'isdigit'  Same error with .isnumber(). Is there a way to check a specific list element if it is a number?

Answer: This could be solved in a one-liner solution, like so: list1 = [1,2,'A','B',3,'4'] list2 = [1,2,3,4,5,6] print([list2[index] for index, x in enumerate(list1) if isinstance(x, int)])  But we can't check if a string can become an int in this specific case. Basically, using list comprehension, we filter the first list and we create a new list based on the second one, so we should implement a way to check if a string can become an int. To do so, we need one more check (and we'll move to a foreach for better readability). list1 = [1,2,'A','B',3,'4'] list2 = [1,2,3,4,5,6] output_list = [] for index, x in enumerate(list1): if isinstance(x, int): output_list.append(list2[index]) if isinstance(x, str): if x.isdigit(): output_list.append(list2[index])  In conclusion, isdigit() is only available for strings, that's why you get the error. You loop through integers too, so it doesn't exist. Note that this code is NOT flexible and it only covers the specific case. For example, if negative numbers are prompted or lists are of different lengths, it will break. If we want to also cover the negative numbers case, a quick fix is to swap from x.isdigit() to a try-except, like so: list1 = [1,2,'A','B',3,'4'] list2 = [1,2,3,4,5,6] output_list = [] for index, x in enumerate(list1): if isinstance(x, int): output_list.append(list2[index]) if isinstance(x, str): try: int(x) output_list.append(list2[index]) except: continue  If it can't be converted to int, then go to the next iteration.
Question: I have just started with the Kucoin API but I'm having trouble executing a market buy order in the futures market using python. I'm using this Github as a reference:  https://github.com/Kucoin/kucoin-futures-python-sdk  heres the code I tried: from kucoin_futures.client import Trade client = Trade(key='myKey', secret='mySecret', passphrase='myPP') client.create_market_order(symbol="ZEROUSDTM", side="buy",funds=1, lever=1)  however this returns the following error: File "C:\Users\User\AppData\Local\Programs\Python\Python312\Lib\site-packages\kucoin_futures\base_request\base_request.py", line 128, in check_response_data raise Exception("{}-{}".format(response_data.status_code, response_data.text)) Exception: 200-{"msg":"Quantity parameter cannot be empty.","code":"100001"}  does anyone know where I'm going wrong? Thanks for reading

Answer: by default orders are limit orders. So you need to specify price and size attributes. This is according the API manual: https://www.kucoin.com/docs/rest/spot-trading/orders/place-order
Question: I have a large dataframe that looks like this: df_large = pl.DataFrame({'x':['h1','h2','h2','h3'], 'y':[1,2,3,4], 'ind1':['0/0','1/0','1/1','0/1'], 'ind2':['0/1','0/2','1/1','0/0'] }).lazy() df_large.collect()  | x | y | ind_1 | ind_2 | |_______|_______|_______|_________| | "h1" | 1 | "0/0" | '0/1' | | "h2" | 2 | "1/0" | '0/2' | | "h2" | 3 | "1/1" | '1/1' | | "h3" | 4 | "0/1" | '0/0' |  df_large contains coordinates [x (str), y (int)] and string values for many individuals [ind_1,ind_2,...]. It is very large, so I have to read the CSV file as a lazy dataframe. Additionally, I have a small dataframe that looks like this: df_rep = pl.DataFrame({'x':['h1','h2','h2'], 'y':[1,2,2], 'ind':['ind1','ind1','ind2']}) df_rep  | x | y | indvs | |_______|_______|_________| | "h1" | 1 | "ind_1" | | "h2" | 2 | "ind_1" | | "h2" | 2 | "ind_2" |  I need to mutate the values for the columns ind_k in df_large when they appears on df_rep. I did the following code for that: for row in df_rep.iter_rows(): df_large = df_large.with_columns( pl.when(pl.col('x') == row[0], pl.col('y') == row[1]) .then(pl.col(row[2]).str.replace_all('(.)/(.)','./.')) .otherwise(pl.col(row[2])) .alias(row[2]) ) df_large.collect()  | x | y | ind_1 | ind_2 | |_______|_______|_______|_________| | "h1" | 1 | "./." | '0/1' | | "h2" | 2 | "./." | './.' | | "h2" | 3 | "1/1" | '1/1' | | "h3" | 4 | "0/1" | '0/0' |  This method, while slow, works for a subset of the larger dataset. However, Polars produces a segmentation fault when applied to the full dataset. I was hoping you could provide feedback on how to resolve this issue. An alternative method to achieve my goal without using iter_rows() would be ideal! I am a beginner with Polars, and I would greatly appreciate any feedback. I've been stuck on this issue for some time now :(

Answer: If you reshape the small frame with .pivot() df_rep.with_columns(value=True).pivot(on="ind", index=["x", "y"])  shape: (2, 4) ┌─────┬─────┬──────┬──────┐ │ x ┆ y ┆ ind1 ┆ ind2 │ │ --- ┆ --- ┆ --- ┆ --- │ │ str ┆ i64 ┆ bool ┆ bool │ ╞═════╪═════╪══════╪══════╡ │ h1 ┆ 1 ┆ true ┆ null │ │ h2 ┆ 2 ┆ true ┆ true │ └─────┴─────┴──────┴──────┘  You could then match the rows with a left .join() and put then when/then logic into a single .with_columns() call. index = ["x", "y"] other = df_rep.with_columns(value=True).pivot(on="ind", index=index) names = other.drop(index).columns (df_large .join(other, on=index, how="left") .with_columns( pl.when(pl.col(f"{name}_right")) .then(pl.col(name).str.replace_all(r"(.)/(.)", "./.")) .otherwise(pl.col(name)) for name in names ) )  shape: (4, 6) ┌─────┬─────┬──────┬──────┬────────────┬────────────┐ │ x ┆ y ┆ ind1 ┆ ind2 ┆ ind1_right ┆ ind2_right │ │ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- │ │ str ┆ i64 ┆ str ┆ str ┆ bool ┆ bool │ ╞═════╪═════╪══════╪══════╪════════════╪════════════╡ │ h1 ┆ 1 ┆ ./. ┆ 0/1 ┆ true ┆ null │ │ h2 ┆ 2 ┆ ./. ┆ ./. ┆ true ┆ true │ │ h2 ┆ 3 ┆ 1/1 ┆ 1/1 ┆ null ┆ null │ │ h3 ┆ 4 ┆ 0/1 ┆ 0/0 ┆ null ┆ null │ └─────┴─────┴──────┴──────┴────────────┴────────────┘  You can then .drop() the names_right columns.
Question: I have a list of time string with different formats as shown time = ["1:5 am", "1:35 am", "8:1 am", "9:14 am", "14:23 pm", "20:2 pm"] dict = {'time': time} df = pd.DataFrame(dict)  and wanted to replace strings in list as shown below. ["01:05 am", "01:35 am", "08:01 am", "09:14 am", "14:23 pm", "20:02 pm"]  Not sure how to write a regex that format the string in DataFrame.

Answer: A possible solution, which is based on regex. (df['time'].str.replace(r'^(\d):', r'0\1:', regex=True) .str.replace(r':(\d)\s', r':0\1 ', regex=True))  The main ideas are:  With r'^(\d):', one matches a single digit at the beginning of the string followed by a colon (e.g., 1: in 1:5 am).  With r'0\1:', one adds a 0 before the captured single digit and retains the colon.  With r':(\d)\s', one matches a single digit after a colon and before a space (e.g., :5 in 1:5 am).  With r':0\1 ', one adds a 0 before the captured single digit and retains the colon and space.   Output: 0 01:05 am 1 01:35 am 2 08:01 am 3 09:14 am 4 14:23 pm 5 20:02 pm Name: time, dtype: object
Question: What I want is to count the filled and blank cells in each column of a .csv file. This is my code: import pandas as pd file_path = r"C:\Users\andre\OneDrive\Documentos\Farmácia\Python\Cadastro_clientes\cadastro_cli.csv" df = pd.read_csv(file_path, sep='|', header=None) # No names argument, read all columns filled_counts = df.count() # Count non-null entries for all columns blank_counts = df.isnull().sum() # Count null (blank) entries for all columns summary = pd.DataFrame({ 'Filled': filled_counts, 'Blank': blank_counts }) print("\nFilled and Blank Counts:") print(summary)  I only get this, which is not what I want at all: Filled and Blank Counts: Filled Blank 0 22318 0  I'm using Jupyter Notebook. Any help or tips are very appreciated!

Answer: If you want to tally the number of filled and blank cells for each column individually, use: summary = pd.DataFrame({ 'Filled': df.notnull().sum(), # Count non-null (filled) cells 'Blank': df.isnull().sum() # Count null (blank) cells }) print(summary, "\n")  If you want to tally for the whole csv, use: total_filled = summary['Filled'].sum() total_blank = summary['Blank'].sum() print(f"Filled: {total_filled}, \nBlank: {total_blank}"  Complete script: import pandas as pd file_path = r"C:\Users\andre\OneDrive\Documentos\Farmácia\Python\Cadastro_clientes\cadastro_cli.csv" df = pd.read_csv(file_path, sep='|') #if csv does not have headers add 'header=None' # Calculate filled and blank counts for each column summary = pd.DataFrame({ 'Filled': df.notnull().sum(), 'Blank': df.isnull().sum() }) print(summary, "\n") # Calculate totals for the entire CSV total_filled = summary['Filled'].sum() total_blank = summary['Blank'].sum() print(f"Total filled: {total_filled} \nTotal blank: {total_blank}")
Question: import requests, re r = requests.get('example.com') p = re.compile('\d') print(p.match(str(r.text)))  This always prints None, even though r.text definitely contains numbers, but print(p.match('12345')) works. What do I need to do to r.text to make it readable by re.compile.match()? Casting to str is clearly insufficient.

Answer: It is because re.match only checks for a match at the beginning of the string, and r.text does not start with a number. If you want to find the first match, then use re.search instead: import requests, re r = requests.get('https://example.com') p = re.compile(r'\d') print(p.search(r.text))  Output: <re.Match object; span=(88, 89), match='8'>  From the docs:  Pattern.match: If zero or more characters at the beginning of string match this regular expression, return a corresponding Match.   Pattern.search: Scan through string looking for the first location where this regular expression produces a match, and return a corresponding Match.
Question: In this code: class A(): def __init__(self, x): self.x = x def __str__(self): return self.x class B(A): def __str__(self): return super().__str__() b = B("Hi") print(b)  The output is: Hi. What is happening under the hood? How does the default constructor in the derived class invoke the super class constructor? How are the params passed to the derived class object get mapped to those of the super class?

Answer:  How does the default constructor in the derived class invoke the super class constructor?  You didn't override it in B, so you inherited it from A. That's what inheritance is for. >>> B.__init__ is A.__init__ True  In the same vein, you may as well not define B.__str__ at all here, since it doesn't do anything (other than add a useless extra frame into the call stack).  How are the params passed to the derived class object get mapped to those of the super class?  You may be overthinking this. As shown above, B.__init__ and A.__init__ are identical. B.__init__ gets resolved in the namespace of A, since it is not present in the namespace of B. >>> B.__mro__ (__main__.B, __main__.A, object) >>> A.__dict__["__init__"] <function __main__.A.__init__(self, x)> >>> B.__dict__["__init__"] ... # KeyError: '__init__'   What is happening under the hood?  Please note that __init__ methods are not constructors. If you're looking for an analogy of constructors in other programming languages, the __new__ method may be more suitable than __init__. First, an instance of B will created (by __new__), and then this instance will be passed as the first positional argument self to A.__init__, along with the string value "Hi" for the second positional argument x.
Question: You have one array of numbers, for example [2, 5, 1]. You have a second array of numbers, for example [8, 4, 3]. For each of the numbers in the second array, how many operations would it take to make the first array all equal that number? You can only increment or decrement by 1 at a time. To get to 8, it would take (8-2)+(8-5)+(8-1)=16 operations. To get to 4, it would take (4-2)+(5-4)+(4-1)=6 operations. To get to 3, it would take (3-2)+(5-3)+(3-1)=5 operations. So the answer would be [16, 6, 5].  I was able to do this in one line: answer = [sum(abs(x-y) for x in a1) for y in a2]  But this wasn't fast enough when the arrays can have up to 105 items. How would I be able to do this faster?

Answer: Same approach as the others (sort, use bisect to split into the i smaller and the n-i larger values, and look up their sums with precomputed prefix sums), just less code: from itertools import accumulate from bisect import bisect def solve(a1, a2): a1.sort() p1 = 0, *accumulate(a1) n = len(a1) total = p1[-1] return [ (i*y - p1[i]) + (total-p1[i] - (n-i)*y) for y in a2 for i in [bisect(a1, y)] ]  Attempt This Online! The expression in the list comprehension could be "optimized" to (2*i-n)*y + total-2*p1[i], but I can't make sense of that and prefer the longer version where I compute the costs for the i smaller values and the n-i larger values separately.
Question: Define a simple method: class Foo: def bar(self): print('bar!')  Now use inspect.ismethod: >>> from inspect import ismethod >>> ismethod(Foo.bar) False  Why is this? Isn't bar a method?

Answer: Consult the documentation:  inspect.ismethod(object) Return True if the object is a bound method written in Python.  So ismethod is actually only testing for bound methods. That means if you create an instance and access the method through the instance, ismethod will return True: >>> obj = Foo() >>> ismethod(obj.bar) True  A "bound" method means a method which is already bound to an object, so that object is provided as the self parameter. This is how you can call obj.bar() with no arguments, even though bar was declared to have one parameter. We can also see the difference by looking at the types: >>> Foo.bar <function Foo.bar at 0x7853771f1ab0> >>> obj.bar <bound method Foo.bar of <__main__.Foo object at 0x7853771e4850>>  Only the latter is a bound method. Foo.bar is a function, it is not considered to be a method by Python's type system.  So why does inspect.ismethod behave this way? From Python's perspective, bar is an ordinary function which happens to be referenced by an attribute of the class Foo. To make this clear, suppose you define a function outside of the class, and then assign it to a class attribute: def baz(self): print('baz!') Foo.baz = baz  If you call ismethod(baz) you should expect it to be False because baz is obviously just a function declared in the outer scope. And you will get the same result if you call ismethod(Foo.baz), because Foo.baz is just a reference to the same function. In fact, the expression Foo.baz is evaluated to get a reference to the function before that reference is passed as an argument to ismethod; of course, ismethod can't give different answers for the same argument depending on where that argument comes from. The same applies when the function bar is declared inside the class Foo. The way classes work in Python, this is pretty much equivalent to declaring the function outside of the class and then assigning it to a class attribute of the same name as the function. So bar and baz work exactly the same way: >>> obj.baz <bound method baz of <__main__.Foo object at 0x7853771e4850>> >>> obj.baz() baz!   In short, Foo.bar is "not a method" because in Python, "methods" are just ordinary functions until they are bound to instances.