eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: For some, simple thread related code, i.e: import threading a = 0 threads = [] def x(): global a for i in range(1_000_000): a += 1 for _ in range(10): thread = threading.Thread(target=x) threads.append(thread) thread.start() for thread in threads: thread.join() print(a) assert a == 10_000_000  We got different behaviour, based on Python version. For 3.10, the output is: ❯ python3.10 b.py 10000000  For 3.9, the output is: ❯ python3.9 b.py 2440951 Traceback (most recent call last): File "/Users/romka/t/threads-test/b.py", line 24, in <module> assert a == 10_000_000 AssertionError  As we do not acquire any lock, for me, results of 3.9 is obvious and expected. Question is why and how 3.10 got "correct" results, while should not? I'm review changelog for Python 3.10 and there is no anything related to threads or GIL which can bring such results.

Answer: An answer from a core developer:  Unintended consequence of Mark Shannon's change that refactors fast opcode dispatching: https://github.com/python/cpython/commit/4958f5d69dd2bf86866c43491caf72f774ddec97 -- the INPLACE_ADD opcode no longer uses the "slow" dispatch path that checks for interrupts and such.
Question: I recently took an interview at a company (starting with M and ending in A) which asked me this question. Still practicing my algorithms, so I was hoping someone could help me understand how to solve this problem, and these types of problems. The problem: You are given 2 arrays. For example: D = [10,7,13,12,4] R = [5,12,7,10,12]  D denotes the departure prices for flights from city A to city B. R denotes the return prices for flights from city B to city A. Find the minimum cost of a round trip flight between city A and city B. For example, the minimum in the example is D[1] + R[2]. (possible only to take the return flight on same or higher index from the departure flight) The tricky part is that, obviously, you must depart before returning. The naïve approach is just a double for loop combining all the possibilities. However, I know there is a better approach, but I can't wrap my head around it. I believe we want to create some sort of temporary array with the minimum so far or something like that... Thanks for reading.

Answer: Create a mono queue/stack out of the return prices array R, then you can solve it in O(n) where n is the length of D. R = [5, 12, 9, 10, 12] => [5, 9, 9, 10, 12]  As you can see at each step we have access to the cheapest return flight that is possible at index i and above. Now, iterate over elements of D and look at that index in monoQ. Since that index at monoQ is the smallest possible in R for i and above, you now that you can't do better at that point. In code: D = [10,7,15,12,4] R = [5,12,9,10,12] monoQ = [0]*len(R) monoQ[-1] = R[-1] for i in range(len(R)-2, -1, -1): monoQ[i] = min(monoQ[i+1], R[i]) best = R[0]+D[0] for i, el in enumerate(D): best = min(best, D[i]+monoQ[i]) print(best)
Question: I use VS Code on a Mac laptop. If I'm using Python I can run the code by pressing the little arrow in the top right,  However, I can't seem to find a keyboard shortcut for this. There is an old question, How to execute Python code from within Visual Studio Code, but all the answers there seem either to be obsolete or not to work on a Mac. One of them says that the F5 key should work, but my Mac has a useless touchbar instead of function keys so it's no help to me. tl;dr is there a shortcut to run Python code on a modern VS Code installation besides F5, or an easy way to set one up?

Answer: I'm using Windows so i can't give you a specific answer. But Code > Preferences > Keyboard Shortcuts, search with keyword run python file, you will get related shortcuts.
Question: I have an issue in python. My original regex expression is: f"regex(metrics_api_failure\.prod\.[\w_]+\.{method_name}\.\d+\.\d+\.[\w_]+\.[\w_]+\.sum\.60)"  (method_name is a local variable) and I got a lint warning: "[FLAKE8 W605] invalid escape sequence '\.'Arc(W605)"  Which looks like recommends me to use r as the regex prefix. But if I do: r"regex(metrics_api_failure\.prod\.[\w_]+\.{method_name}\.\d+\.\d+\.[\w_]+\.[\w_]+\.sum\.60)"  The {method_name} becomes the string type rather than a passed in variable. Does anyone have an idea how to solve this dilemma?

Answer: Pass in the expression: r"regex(metrics_api_failure\.prod\.[\w_]+\." + method_name + r"\.\d+\.\d+\.[\w_]+\.[\w_]+\.sum\.60)"  Essentially, use Python string concatenation to accomplish the same thing that you were doing with the brackets. Then, r"" type string escaping should work. or use a raw format string: rf"regex(metrics_api_failure\.prod\.[\w_]+\.{method_name}\.\d+\.\d+\.[\w_]+\.[\w_]+\.sum\.60)"
Question: In the following code, I create two lists with the same values: one list unsorted (s_not), the other sorted (s_yes). The values are created by randint(). I run some loop for each list and time it. import random import time for x in range(1,9): r = 10**x # do different val for the bound in randint() m = int(r/2) print("For rand", r) # s_not is non sorted list s_not = [random.randint(1,r) for i in range(10**7)] # s_yes is sorted s_yes = sorted(s_not) # do some loop over the sorted list start = time.time() for i in s_yes: if i > m: _ = 1 else: _ = 1 end = time.time() print("yes", end-start) # do the same to the unsorted list start = time.time() for i in s_not: if i > m: _ = 1 else: _ = 1 end = time.time() print("not", end-start) print()  With output: For rand 10 yes 1.0437555313110352 not 1.1074268817901611 For rand 100 yes 1.0802974700927734 not 1.1524150371551514 For rand 1000 yes 2.5082249641418457 not 1.129960298538208 For rand 10000 yes 3.145440101623535 not 1.1366300582885742 For rand 100000 yes 3.313387393951416 not 1.1393756866455078 For rand 1000000 yes 3.3180911540985107 not 1.1336982250213623 For rand 10000000 yes 3.3231537342071533 not 1.13503098487854 For rand 100000000 yes 3.311596393585205 not 1.1345293521881104  So, when increasing the bound in the randint(), the loop over the sorted list gets slower. Why?

Answer: Cache misses. When N int objects are allocated back-to-back, the memory reserved to hold them tends to be in a contiguous chunk. So crawling over the list in allocation order tends to access the memory holding the ints' values in sequential, contiguous, increasing order too. Shuffle it, and the access pattern when crawling over the list is randomized too. Cache misses abound, provided there are enough different int objects that they don't all fit in cache. At r==1, and r==2, CPython happens to treat such small ints as singletons, so, e.g., despite that you have 10 million elements in the list, at r==2 it contains only (at most) 100 distinct int objects. All the data for those fit in cache simultaneously. Beyond that, though, you're likely to get more, and more, and more distinct int objects. Hardware caches become increasingly useless then when the access pattern is random. Illustrating: >>> from random import randint, seed >>> seed(987987987) >>> for x in range(1, 9): ... r = 10 ** x ... js = [randint(1, r) for _ in range(10_000_000)] ... unique = set(map(id, js)) ... print(f"{r:12,} {len(unique):12,}") ... 10 10 100 100 1,000 7,440,909 10,000 9,744,400 100,000 9,974,838 1,000,000 9,997,739 10,000,000 9,999,908 100,000,000 9,999,998
Question: I'm a little confuse how I'm supposed to type a base class abstract method? In this case my base class only requires that the inheriting class implements a method named 'learn' that returns None without mandating any arguments. class MyBaseClass(ABC): @abstractmethod def learn(self, *args, **kwargs) -> None: raise NotImplementedError()  but if I implement it mypy raise en error 'Signature of "learn" incompatible with supertype "MyBaseClass"' class MyOtherClass(MyBaseClass): def learn(self, alpha=0.0, beta=1) -> None: # do something return None  So how should I declare the learn method in the base class?

Answer: First things first, I'd ask why you want a method without known arguments. That sounds like a design problem. One solution It's fine to add new parameters to subclasses if those parameters have default values (and the base class doesn't use **kwargs), like class MyBaseClass: @abstractmethod def learn(self) -> None: raise NotImplementedError() class MyOtherClass(MyBaseClass): def learn(self, alpha=0.0, beta=1) -> None: ...  though you won't be able to specify alpha and beta if you only know it's a MyBaseClass: def foo(x: MyOtherClass) -> None: x.learn(alpha=3) # ok def foo(x: MyBaseClass) -> None: x.learn(alpha=3) # not ok  Why didn't *args, **kwargs work? If you have class MyBaseClass(ABC): @abstractmethod def learn(self, *args, **kwargs) -> None: raise NotImplementedError()  then consider the function def foo(x: MyBaseClass) -> None: ...  In that function, you can pass anything to x.learn, for example x.learn(1, fish='three'). This has to be true for any x. Since x can be an instance of a subclass of MyBaseClass (such as MyOtherClass), that subclass must also be able to accept those arguments.
Question: I have a django project using easy-thumbnail as a dependency. Installing all packages with pip is working as expected, but when I try to run my app I get this error: Invalid template library specified. ImportError raised when trying to load 'backend.templatetags.get_thumbnail': dlopen(/opt/homebrew/lib/python3.9/site-packages/reportlab/graphics/_renderPM.cpython-39-darwin.so, 0x0002): symbol not found in flat namespace '_FT_Done_Face'  The error is raised from reportlab which is a dependency of easy-thumbnail. As far as I understand, reportlab is not able to find freetype. But it is installed correctly imho. I'm using macOS 12.0.1 I installed Python and freetype via Homebrew. pkg-config says, that freetype2 is available at the expected paths. What am I doing wrong? How can I fix this?  Edit I did otool -l on the failing .so-file and this is, what I get (here I'm running it again in a venv): /Users/markusgerards/.pyenv/versions/myapp/lib/python3.9/site-packages/reportlab/graphics/_renderPM.cpython-39-darwin.so: /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1292.60.1)  I suspect that freetype should be listed there... right?

Answer: I reinstalled reportlab with this command: pip install reportlab --force-reinstall --no-cache-dir --global-option=build_ext  This forced pip to actually build the package and now everythings works as intended!
Question: I have this model : # Set random seed tf.random.set_seed(42) # Create some regression data X_regression = np.expand_dims(np.arange(0, 1000, 5), axis=0) y_regression = np.expand_dims(np.arange(100, 1100, 5), axis=0) # Split it into training and test sets X_reg_train = X_regression[:150] X_reg_test = X_regression[150:] y_reg_train = y_regression[:150] y_reg_test = y_regression[150:]  # Setup random seed tf.random.set_seed(42) # Recreate the model model_3 = tf.keras.Sequential([ tf.keras.layers.Dense(100), tf.keras.layers.Dense(10), tf.keras.layers.Dense(1) ]) # Change the loss and metrics of our compiled model model_3.compile(loss=tf.keras.losses.mae, # change the loss function to be regression-specific optimizer=tf.keras.optimizers.Adam(learning_rate=0.01), metrics=['mae']) # change the metric to be regression-specific # Fit the recompiled model model_3.fit(X_reg_train, y_reg_train, epochs=100)  To begin with, the model does not train well  To add on, when I try to predict using that model, I get the following error :  Why am I getting the above error and how can I fix it?

Answer: Change the axis dimension in expand_dims to 1 and slice your data like this, since it is 2D: import tensorflow as tf import numpy as np tf.random.set_seed(42) # Create some regression data X_regression = np.expand_dims(np.arange(0, 1000, 5), axis=1) y_regression = np.expand_dims(np.arange(100, 1100, 5), axis=1) # Split it into training and test sets X_reg_train = X_regression[:150, :] X_reg_test = X_regression[150:, :] y_reg_train = y_regression[:150, :] y_reg_test = y_regression[150:, :] tf.random.set_seed(42) # Recreate the model model_3 = tf.keras.Sequential([ tf.keras.layers.Dense(100), tf.keras.layers.Dense(10), tf.keras.layers.Dense(1) ]) # Change the loss and metrics of our compiled model model_3.compile(loss=tf.keras.losses.mae, # change the loss function to be regression-specific optimizer=tf.keras.optimizers.Adam(learning_rate=0.01), metrics=['mae']) # change the metric to be regression-specific # Fit the recompiled model model_3.fit(X_reg_train, y_reg_train, epochs=100) model_3.predict(X_reg_test)
Question: I have a dataframe that looks like this: D X Y Z A 22 16 23 A 21 16 22 A 20 17 21 B 33 50 11 B 34 53 12 B 34 55 13 C 44 34 11 C 45 33 11 C 45 33 10 D 55 35 60 D 57 34 61 E 66 36 13 E 67 38 14 E 67 37 13  I want to get the minimum and maximum values of the categorical variable D across all the column values and so the output dataframe should look something like this: D Xmin Xmax Ymin Ymax Zmin Zmax A 20 22 16 17 21 23 B 33 34 50 55 11 13 C 44 45 33 34 10 11 D 55 57 34 35 60 61 E 66 67 36 38 13 14  I have tried this, but no luck: min_max_df = dfObj.groupby('D').agg({'X': [dfObj.min(axis=0), dfObj.max(axis=0)]}) 

Answer: I believe this is a nice way of doing it and in a single line of code. Making use of join doing the operation by index and the rsuffix and lsuffix to differentiate min and max. output = df.groupby('D').min().join(df.groupby('D').max(), lsuffix='min', rsuffix='max')  Outputs:  Xmin Xmax Ymin Ymax Zmin Zmax D A 20 22 16 17 21 23 B 33 34 50 55 11 13 C 44 45 33 34 10 11 D 55 57 34 35 60 61 E 66 67 36 38 13 14
Question: I am looking for an easy way to count the number of the green pixels in the image below, where the original image is the same but the green pixels are black. I tried it with numpy.diff(), but then I am counting some pixels twice. I thought about numpy.gradient() – but here I am not sure if it is the right tool. I know there have to be many solutions to this problem, but I don't know how to google for it. I am looking for a solution in python.  To make it clearer, I have only one image (only black and white pixels). The image with the green pixel is just for illustration.

Answer: You can use the edge detection kernel for this problem. import numpy as np from scipy.ndimage import convolve a = np.array([[0, 0, 0, 0], [0, 1, 1, 1], [0, 1, 1, 1]]) kernel = np.array([[-1, -1, -1], [-1, 8, -1], [-1, -1, -1]])  Then, we will convolve the original array with the kernel. Notice that the edges are all negatives. >>> convolve(a, kernel) [[-1 -2 -3 -3] [-2 5 3 3] [-3 3 0 0]]  We will count the number of negative values and get the result. >>> np.where(convolve(a, kernel) < 0, 1, 0) [[1 1 1 1] [1 0 0 0] [1 0 0 0]] >>> np.sum(np.where(convolve(a, kernel) < 0, 1, 0)) 6   Edges-only kernel There are a lot of things you can do with the kernel. For example, you can modify the kernel if you don't want to include diagonal neighbors. kernel = np.array([[ 0, -1, 0], [-1, 4, -1], [ 0, -1, 0]])  This gives the following output. >>> np.where(convolve(a, kernel) < 0, 1, 0) [[0 1 1 1] [1 0 0 0] [1 0 0 0]] >>> np.sum(np.where(convolve(a, kernel) < 0, 1, 0)) 5
Question: I have a dataset like the below:     epoch_seconds eq_time     1636663343887 2021-11-12 02:12:23     Now, I am trying to convert the eq_time to epoch seconds which should match the value of the first column but am unable to do so. Below is my code: df = spark.sql("select '1636663343887' as epoch_seconds") df1 = df.withColumn("eq_time", from_unixtime(col("epoch_seconds") / 1000)) df2 = df1.withColumn("epoch_sec", unix_timestamp(df1.eq_time)) df2.show(truncate=False)  I am getting output like below:     epoch_seconds eq_time epoch_sec     1636663343887 2021-11-12 02:12:23 1636663343     I tried this link as well but didn't help. My expected output is that the first and third columns should match each other. P.S: I am using the Spark 3.1.1 version on local whereas it is Spark 2.4.3 in production, and my end goal would be to run it in production.

Answer: Use to_timestamp instead of from_unixtime to preserve the milliseconds part when you convert epoch to spark timestamp type. Then, to go back to timestamp in milliseconds, you can use unix_timestamp function or by casting to long type, and concatenate the result with the fraction of seconds part of the timestamp that you get with date_format using pattern S: import pyspark.sql.functions as F df = spark.sql("select '1636663343887' as epoch_ms") df2 = df.withColumn( "eq_time", F.to_timestamp(F.col("epoch_ms") / 1000) ).withColumn( "epoch_milli", F.concat(F.unix_timestamp("eq_time"), F.date_format("eq_time", "S")) ) df2.show(truncate=False) #+-------------+-----------------------+-------------+ #|epoch_ms |eq_time |epoch_milli | #+-------------+-----------------------+-------------+ #|1636663343887|2021-11-11 21:42:23.887|1636663343887| #+-------------+-----------------------+-------------+
Question: I'm wondering if there's any simpler way to assign multiple columns in Python, just like the := in R data.table. For example, in Python I would have to write like this: df['Col_A'] = df.A/df.B df['Col_B'] = df.C/df.D df['Col_C'] = df.E/df.F * 1000000 df['Col_D'] = df.G/df.H * 1000000  However, it's just one line of code in R data.table: df[, ':='(Col_A = A/B, Col_B = C/D, Col_C = E/F*1000000, Col_B = G/H*1000000)] 

Answer: You can use DataFrame.assign to assign multiple columns: df = df.assign(Col_A=df.A/df.B, Col_B=df.C/df.D, Col_C=df.E/df.F*1000000, Col_D=df.G/df.H*1000000)  Example: df = pd.DataFrame(np.random.random((4, 8)), columns=list('ABCDEFGH')) # A B ... H # 0 0.771211 0.238201 ... 0.311904 # 1 0.197548 0.635218 ... 0.626639 # 2 0.332333 0.838589 ... 0.477978 # 3 0.929690 0.327412 ... 0.046179  df = df.assign(Col_A=df.A/df.B, Col_B=df.C/df.D, Col_C=df.E/df.F*1000000, Col_D=df.G/df.H*1000000) # A B ... H Col_A Col_B Col_C Col_D # 0 0.771211 0.238201 ... 0.311904 3.237647 1.547285 1.463586e+06 2.845234e+06 # 1 0.197548 0.635218 ... 0.626639 0.310993 1.385892 1.394466e+07 2.685293e+05 # 2 0.332333 0.838589 ... 0.477978 0.396300 0.078238 8.494174e+06 6.001031e+05 # 3 0.929690 0.327412 ... 0.046179 2.839514 0.852443 1.962892e+06 8.791233e+06   If you want column names with spaces, you can use a dict: df = df.assign(**{'Col A': df.A/df.B, 'Col B': df.C/df.D, 'Col C': df.E/df.F*1000000, 'Col D': df.G/df.H*1000000}) # A B ... H Col A Col B Col C Col D # 0 0.868320 0.086743 ... 0.505330 10.010311 6.680195 1.147554e+06 2.620416e+05 # 1 0.244341 0.908793 ... 0.389684 0.268863 2.388179 2.196769e+06 2.235063e+06 # 2 0.917949 0.248149 ... 0.710027 3.699188 0.453094 1.311617e+06 1.004200e+06 # 3 0.616655 0.498817 ... 0.703579 1.236235 2.186589 1.272981e+06 8.602272e+05
Question: I have a DataFrame:      name age     0 Paul 25   1 John 27   2 Bill 23     I know that if I enter:  df[['name']] = df[['age']]   I'll get the following:      name age     0 25 25   1 27 27   2 23 23     But I expect the same result from the command:  df.loc[:, ['name']] = df.loc[:, ['age']]   But instead, I get this:      name age     0 NaN 25   1 NaN 27   2 NaN 23     For some reason, if I omit those square brackets [] around column names, I'll get exactly what I expected. That is the command:  df.loc[:, 'name'] = df.loc[:, 'age']   gives the right result:      name age     0 25 25   1 27 27   2 23 23     Why does two pairs of brackets with .loc result in NaN? Is it some sort of a bug or is it intended behaviour? I can't figure out the reason for such a behaviour.

Answer: That's because for the loc assignment all index axes are aligned, including the columns: Since age and name do not match, there is no data to assign, hence the NaNs. You can make it work by renaming the columns: df.loc[:, ["name"]] = df.loc[:, ["age"]].rename(columns={"age": "name"})  or by accessing the numpy array: df.loc[:, ["name"]] = df.loc[:, ["age"]].values
Question: I'm having problems with installing a package using pip. I tried: pip install jurigged  Causing these errors: ERROR: Could not find a version that satisfies the requirement jurigged (from versions: none) ERROR: No matching distribution found for jurigged  I checked if pip was up to date which was the case. I'm on Python 3.7.4. Does anyone know a solution to this problem?

Answer: From PyPI, jurigged is only supported as of Python >= 3.8 (see also here) pip doesn't find anything to install because you do not meet the requirements. Upgrade to Python >= 3.8 and do the same: pip install jurigged
Question: I am using HuggingFace transformers AutoTokenizer to tokenize small segments of text. However this tokenization is splitting incorrectly in the middle of words and introducing # characters to the tokens. I have tried several different models with the same results. Here is an example of a piece of text and the tokens that were created from it. CTO at TLR Communications Pty Ltd ['[CLS]', 'CT', '##O', 'at', 'T', '##LR', 'Communications', 'P', '##ty', 'Ltd', '[SEP]']  And here is the code I am using to generate the tokens tokenizer = AutoTokenizer.from_pretrained("tokenizer_bert.json") tokens = tokenizer.tokenize(tokenizer.decode(tokenizer.encode(sequence))) 

Answer: This is not an error but a feature. BERT and other transformers use WordPiece tokenization algorithm that tokenizes strings into either: (1) known words; or (2) "word pieces" for unknown words in the tokenizer vocabulary. In your examle, words "CTO", "TLR", and "Pty" are not in the tokenizer vocabulary, and thus WordPiece splits them into subwords. E.g. the first subword is "CT" and another part is "##O" where "##" denotes that the subword is connected to the predecessor. This is a great feature that allows to represent any string.
Question: Simple problem that has me completely dumbfounded. I am trying to read an Excel document with pandas but I am stuck with this error:  ValueError: Worksheet index 0 is invalid, 0 worksheets found  My code snippet works well for all but one Excel document linked below. Is this an issue with my Excel document (which definitely has sheets when I open it in Excel) or am I missing something completely obvious? Excel Document EDIT - Forgot the code. It is quite simply: import pandas as pd df = pd.read_excel(FOLDER + 'omx30.xlsx')  FOLDER Is the absolute path to the folder in which the file is located.

Answer: It seems there indeed is a problem with my excel file. We have not been able to figure out what though. For now the path of least resistance is simply saving as a .csv in excel and using pd.read_csv to read this instead.