eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I'd like to write some unit tests for simple methods which have pyspark code. def do_stuff(self, df1: DataFrame, df2_path: str, df1_key: str, df2_key: str) -> DataFrame: df2 = self.spark.read.format('parquet').load(df2_path) return df1.join(df2, [f.col(df1_key) == f.col(df2_key)], 'left')  How can I mock the spark read part? I've tried this: @patch("class_to_test.SparkSession") def test_do_stuff(self, mock_spark: MagicMock) -> None: spark = MagicMock() spark.read.return_value.format.return_value.load.return_value = \ self.spark.createDataFrame([(1, 2)], ["key2", "c2"]) mock_spark.return_value = spark input_df = self.spark.createDataFrame([(1, 1)], ["key1", "c1"]) actual_df = ClassToTest().do_stuff(input_df, "df2", "key1", "key2") expected_df = self.spark.createDataFrame([(1, 1, 1, 2)], ["key1", "c1", "key2", "c2"]) assert_pyspark_df_equal(actual_df, expected_df)  But it fails with this error: py4j.Py4JException: Method join([class java.util.ArrayList, class org.apache.spark.sql.Column, class java.lang.String]) does not exist Looks like the mocking didn't work as I expected, what should I do with it so the spark.read.load returns the test dataframe that I specified? Edit: full code here

Answer: You can do it using PropertyMock. Here is an example: # test.py import unittest from unittest.mock import patch, PropertyMock, Mock from pyspark.sql import SparkSession, DataFrame, functions as f from pyspark_test import assert_pyspark_df_equal class ClassToTest: def __init__(self) -> None: self._spark = SparkSession.builder.getOrCreate() @property def spark(self): return self._spark def do_stuff(self, df1: DataFrame, df2_path: str, df1_key: str, df2_key: str) -> DataFrame: df2 = self.spark.read.format('parquet').load(df2_path) return df1.join(df2, [f.col(df1_key) == f.col(df2_key)], 'left') class TestClassToTest(unittest.TestCase): def setUp(self) -> None: self.spark = SparkSession.builder.getOrCreate() def test_do_stuff(self) -> None: # let's say ClassToTest().spark.read.format().load() will return a DataFrame with patch( # change __main__ to your module... '__main__.ClassToTest.spark', new_callable=PropertyMock, return_value=Mock( # read property read=Mock( # format() method format=Mock( return_value=Mock( # load() method result: load=Mock(return_value=self.spark.createDataFrame([(1, 2)], ['key2', 'c2'])))))) ): input_df = self.spark.createDataFrame([(1, 1)], ['key1', 'c1']) df = ClassToTest().do_stuff(input_df, 'df2_path', 'key1', 'key2') assert_pyspark_df_equal( df, self.spark.createDataFrame([(1, 1, 1, 2)], ['key1', 'c1', 'key2', 'c2']) ) if __name__ == '__main__': unittest.main()  Let's check: python test.py # result: ---------------------------------------------------------------------- Ran 1 test in 7.460s OK
Question: Im having trouble in understanding why a post from a third party site is being rejected even though the site is added to CSRF_TRUSTED_ORIGINS list in settings.py. Im receiving a 403 error after the post stating the the csrf check has failed. I thought that adding the site to CSRF_TRUSTED_ORIGINS should make the site exempt from csrf checks. Is there something else I should have done in order to receive post requests from external origins? Im running django 3.2 CSRF_TRUSTED_ORIGINS = ['site.lv']  request headers: Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9 Accept-Encoding: gzip, deflate, br Accept-Language: en-US,en;q=0.9,lv;q=0.8,ru;q=0.7 Cache-Control: no-cache Connection: keep-alive Content-Length: 899 Content-Type: application/x-www-form-urlencoded Host: cvcentrs-staging.herokuapp.com Origin: https://www.site.lv Pragma: no-cache Referer: https://www.site.lv/ sec-ch-ua: " Not A;Brand";v="99", "Chromium";v="96", "Microsoft Edge";v="96" sec-ch-ua-mobile: ?0 sec-ch-ua-platform: "Windows" Sec-Fetch-Dest: document Sec-Fetch-Mode: navigate Sec-Fetch-Site: cross-site Upgrade-Insecure-Requests: 1 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36 Edg/96.0.1054.62 

Answer: This assumption is wrong:  I thought that adding the site to CSRF_TRUSTED_ORIGINS should make the site exempt from csrf checks.  Adding the URL to CSRF_TRUSTED_ORIGINS is only one thing you need to do to allow a POST request from a form on an external domain. You also need to:  Make an AJAX-call from the external page to get a csrf_token, and send the token with your POST request. Set up CORS (see this article for example: https://www.stackhawk.com/blog/django-cors-guide/).  Starting from Django 4.0, you have to include the scheme in CSRF_TRUSTED_ORIGINS: CSRF_TRUSTED_ORIGINS = ['https://site.lv', 'https://www.site.lv']
Question: I want to apply a partial tucker decomposition algorithm to minimize MNIST image tensor dataset of (60000,28,28), in order to conserve its features when applying another machine algorithm afterwards like SVM. I have this code that minimizes the second and third dimension of the tensor i = 16 j = 10 core, factors = partial_tucker(train_data_mnist, modes=[1,2],tol=10e-5, rank=[i,j]) train_datapartial_tucker = tl.tenalg.multi_mode_dot(train_data_mnist, factors, modes=modes, transpose=True) test_data_partial_tucker = tl.tenalg.multi_mode_dot(test_data_mnist, factors, modes=modes, transpose=True)  How to find the best rank [i,j] when I'm using partial_tucker in tensorly that will give the best dimension reduction for the image while conserving as much data?

Answer: Just like principal component analysis the partial tucker decomposition will give better results as we increase the rank, in the sense that the optimal mean square residual of the reconstruction is smaller. In general, features (the core tensor) that enables accurate reconstructions of the original data, can be used to make similar predictions (given any model we can prepend a transformation that reconstruct the original data from the core features). import mxnet as mx import numpy as np import tensorly as tl import matplotlib.pyplot as plt import tensorly.decomposition # Load data mnist = mx.test_utils.get_mnist() train_data = mnist['train_data'][:,0] err = np.zeros([28,28]) # here I will save the errors for each rank batch = train_data[::100] # process only 1% of the data to go faster for i in range(1,28): for j in range(1,28): if err[i,j] == 0: # Decompose the data core, factors = tl.decomposition.partial_tucker( batch, modes=[1,2], tol=10e-5, rank=[i,j]) # Reconstruct data from features c = tl.tenalg.multi_mode_dot(core, factors, modes=[1,2]); # Calculate the RMS error and save err[i,j] = np.sqrt(np.mean((c - batch)**2)); # Plot the statistics plt.figure(figsize=(9,6)) CS = plt.contour(np.log2(err), levels=np.arange(-6, 0)); plt.clabel(CS, CS.levels, inline=True, fmt='$2^{%d}$', fontsize=16) plt.xlabel('rank 2') plt.ylabel('rank 1') plt.grid() plt.title('Reconstruction RMS error');   Usually you have better result with a balanced rank, i.e. i and j not very different from each other. As we increase the error we can get better compression, we can rank the (i,j) by the error, and plot only where the error is minimum for a given feature dimension i * j, like this X = np.zeros([28, 28]) X[...] = np.nan; p = 28 * 28; for e,i,j in sorted([(err[i,j], i, j) for i in range(1, 28) for j in range(1, 28)]): if p < i * j: # we can achieve this error with some better compression pass else: p = i * j; X[i,j] = e; plt.imshow(X)   Anywhere in the white region you are wasting resources, the choice
Question: I am working with a dataframe in Pandas and I need a solution to automatically modify one of the columns that has duplicate values. It is a column type 'object' and I would need to modify the name of the duplicate values. The dataframe is the following:  City Year Restaurants 0 New York 2001 20 1 Paris 2000 40 2 New York 1999 41 3 Los Angeles 2004 35 4 Madrid 2001 22 5 New York 1998 33 6 Barcelona 2001 15  As you can see, New York is repeated 3 times. I would like to create a new dataframe in which this value would be automatically modified and the result would be the following:  City Year Restaurants 0 New York 2001 2001 20 1 Paris 2000 40 2 New York 1999 1999 41 3 Los Angeles 2004 35 4 Madrid 2001 22 5 New York 1998 1998 33 6 Barcelona 2001 15  I would also be happy with "New York 1", "New York 2" and "New York 3". Any option would be good.

Answer: Use np.where, to modify column City if duplicated df['City']=np.where(df['City'].duplicated(keep=False), df['City']+' '+df['Year'].astype(str),df['City'])
Question: I'm trying to install Openvino to convert a Keras model into a representation for the inference engine. I'm running the command: python3 openvino/tools/mo/mo_tf.py —model_13.h5/ --input_shape=\[180,180\]  This returns the error: from openvino.tools.mo.subprocess_main import subprocess_main ModuleNotFoundError: No module named 'openvino'  I've tried pip install openvino but consistently get: ERROR: Could not find a version that satisfies the requirement openvino (from versions: none) ERROR: No matching distribution found for openvino  To try and make sure the versions of python for running the script and installing Openvino are the same, I've tried: python3 -m pip install openvino  The content of the mo_tf.py script is simply: #!/usr/bin/env python3 # Copyright (C) 2018-2021 Intel Corporation # SPDX-License-Identifier: Apache-2.0 if __name__ == "__main__": from openvino.tools.mo.subprocess_main import subprocess_main subprocess_main(framework='tf')  Has anyone seen this issue and found a workaround?

Answer: The latest version of openvino is 2021.4.2. The list of packages to download by pip includes packages for Python 3.6-3.9 for Linux, MacOS on Intel, and Windows; only packages for 64-bit platforms are provided. No packages for Python 3.10 and no source code. The solution is either to compile from sources, or install with Docker or install from Anaconda. Or downgrade to Python 3.9.
Question: How should I type hint in Python a pandas DataFrameGroupBy object? Should I just use pd.DataFrame as for normal pandas dataframes? I didn't find any other solution atm

Answer: DataFrameGroupBy is a proper type in of itself. So if you're writing a function which must specifically take a DataFrameGroupBy instance: from pandas.core.groupby import DataFrameGroupBy def my_function(dfgb: DataFrameGroupBy) -> None: """Do something with dfgb."""  If you're looking for a more general polymorphic type, there are several possibilities:  pandas.core.groupby.GroupBy since DataFrameGroupBy inherits from GroupBy[DataFrame]. If you want to accept Series instances too, you could either union DataFrameGroupBy and SeriesGroupBy or you could use GroupBy[FrameOrSeries] (if you intend to always match the input type in your return value) or GroupBy[FrameOrSeriesUnion] if your output type doesn't reflect the input type. All of these types are in pandas.core.groupby.generic. You could combine the above generics (and others) in many different ways to your liking.
Question: I have a list that looks as follows (the 'None' in the list is a string, not a None): profit = [1 , 20 , 3 , 5 , 90 , 'None', 900, 67 , 'None'] name = ['a', 'b', 'c', 'e', 'd', 'f' , 'g', 'k', 'pp']  The profit list is a list of "profit" values, so I had to sort it in a reversed order so that the highest values will be at the beginning. Also, I have more lists with the same length that represent other things that are connected to the profit list (for example the name list that shows where the profit came from). Now, I wrote the following code to sort the profit list in a reversed order, and I save the indices so I could sort the other lists (like name) according to the obtained indices: sorted_ind = sorted(range(len(profit)), key=lambda k: profit[k], reverse=True) for i in sorted_ind: print('{0:^50}|{1:^7}|'.format(profit[i], name[I]))  The above code works great when my profit list contains only numbers. However, there are cases where I have no profit and I would like this value to be 'None' (I don't want to set it to 0). I'm trying to perform the same sort but in a way that all of the None indices will be at the end of the list - to sort all the ints and insert the Nones at the end. Any good way to do it?

Answer: You can use a "priority" key to the sort function which checks the types as well: >>> sorted(profit, key=lambda x: (isinstance(x, int), x), reverse=True) [900, 90, 67, 20, 5, 3, 1, 'None', 'None']   If you're doing that just to sort the names list, then it is not necessary to sort the indices, you should use zip: profits = [1, 20, 3, 5, 90, 'None', 900, 67, 'None'] names = ['a', 'b', 'c', 'e', 'd', 'f', 'g', 'k', 'pp'] sorted_ind = sorted(zip(profits, names), key=lambda tup: (isinstance(tup[0], int), tup[0]), reverse=True) for profit, name in sorted_ind: print('{0:^50}|{1:^7}|'.format(profit, name))  Gives:  900 | g | 90 | d | 67 | k | 20 | b | 5 | e | 3 | c | 1 | a | None | f | None | pp |
Question: I am new to queue & threads kindly help with the below code , here I am trying to execute the function hd , I need to run the function multiple times but only after a single run has been completed import queue import threading import time fifo_queue = queue.Queue() def hd(): print("hi") time.sleep(1) print("done") for i in range(3): cc = threading.Thread(target=hd) fifo_queue.put(cc) cc.start()  Current Output hi hi hi donedonedone  Expected Output hi done hi done hi done​ 

Answer: You can use a Semaphore for your purposes  A semaphore manages an internal counter which is decremented by each acquire() call and incremented by each release() call. The counter can never go below zero; when acquire() finds that it is zero, it blocks, waiting until some other thread calls release().  A default value of Semaphore is 1,  class threading.Semaphore(value=1)  so only one thread would be active at once: import queue import threading import time fifo_queue = queue.Queue() semaphore = threading.Semaphore() def hd(): with semaphore: print("hi") time.sleep(1) print("done") for i in range(3): cc = threading.Thread(target=hd) fifo_queue.put(cc) cc.start()  hi done hi done hi done   As @user2357112supportsMonica mentioned in comments RLock would be more safe option  class threading.RLock   This class implements reentrant lock objects. A reentrant lock must be released by the thread that acquired it. Once a thread has acquired a reentrant lock, the same thread may acquire it again without blocking; the thread must release it once for each time it has acquired it.  import queue import threading import time fifo_queue = queue.Queue() lock = threading.RLock() def hd(): with lock: print("hi") time.sleep(1) print("done") for i in range(3): cc = threading.Thread(target=hd) fifo_queue.put(cc) cc.start()
Question: I'm new to numpy and I'm currently working on a modeling project for which I have to perform some calculations based on two different data sources. However until now I haven't managed to figure out how I could multiply all the individual values to each other: I have two data frames One 2D-dataframe: df1 = np.array([[1, 2, 3], [4, 5, 6], [7, 8, 9]])  One 1D-dataframe: df2 = np.array([1, 2, 3, 4, 5])  I would like to multiply all the individual values within the first dataframe (df1) separately with all the values that are stored within the second dataframe in order to create a data cube / new 3D-dataframe that has the shape 5x3x3: df3 = np.array([[[1, 2, 3], [4, 5, 6], [7, 8, 9]], [[2, 4, 6], [8, 10, 12], [14, 16, 18]], ..... ])  I tried different methods but every time I failed to obtain something that looks like df3. x = np.array([[1, 2, 3], [4, 5, 6], [7, 8, 9]]) y = np.array([1, 2, 3, 4, 5]) z = y for i in range(len(z)): z.iloc[i] = x for i in range(0, 5): for j in range(0, 3): for k in range(0, 3): z.iloc[i, j, k] = y.iloc[i] * x.iloc[j, k] print(z)  Could someone help me out with some example code? Thank you!

Answer: Try this: df1 = np.array([[1, 2, 3], [4, 5, 6], [7, 8, 9]]) df2 = np.array([1, 2, 3, 4, 5]) df3 = df1 * df2[:, None, None]  Output: >>> df3 array([[[ 1, 2, 3], [ 4, 5, 6], [ 7, 8, 9]], [[ 2, 4, 6], [ 8, 10, 12], [14, 16, 18]], [[ 3, 6, 9], [12, 15, 18], [21, 24, 27]], [[ 4, 8, 12], [16, 20, 24], [28, 32, 36]], [[ 5, 10, 15], [20, 25, 30], [35, 40, 45]]])
Question: When displaying summary_plot, the color bar does not show. shap.summary_plot(shap_values, X_train)   I have tried changing plot_size. When the plot is higher the color bar appears, but it is very small - doesn't look like it should. shap.summary_plot(shap_values, X_train, plot_size=0.7)   Here is an example of a proper looking color bar.  Does anyone know if this can be fixed somehow? How to reproduce: import pandas as pd import shap import sklearn from sklearn.ensemble import RandomForestRegressor # a classic housing price dataset X,y = shap.datasets.boston() # a simple linear model model = RandomForestRegressor(max_depth=6, random_state=0, n_estimators=10) model.fit(X, y) shap_values = shap.TreeExplainer(model).shap_values(X) shap.summary_plot(shap_values, X)  In this case, the color bar is displayed, but it is very small. I have chosen such an example to make it easy to retrieve the data.

Answer: I had the same problem as you did, and I found that the solution was to downgrade matplotlib to 3.4.3.. It appears SHAP isn't optimized for matplotlib 3.5.1 yet.
Question: The function above  AllPaths() appends an array containing the path to each leaf of the binary tree to the global array  res. The code works just fine, but I want to remove the global variable  res and make the function return an array instead. How can I do that? class Node: def __init__(self, value, left=None, right=None) -> None: self.value = value self.left = left self.right = right res = [] def allPaths(node, arr=[]): if node: tmp = [*arr, node.value] if not node.left and not node.right: # Leaf res.append(tmp) allPaths(node.left, tmp) allPaths(node.right, tmp) root = Node(1) root.left = Node(2); root.left.left = Node(4); root.left.right = Node(5); root.right = Node(3); root.right.right = Node(6); """ 1 <-- root / \ 2 3 / \ \ 4 5 6 <-- leaves """ allPaths(root) print(res) # Output : [[1, 2, 4], [1, 2, 5], [1, 3, 6]] 

Answer: A simple way that allow you to avoid the inner lists and global list altogether is to make a generator that yields the values as they come. Then you can just pass this to list to make the final outcome: class Node: def __init__(self, value, left=None, right=None) -> None: self.value = value self.left = left self.right = right def allPaths(node): if node: if not node.left and not node.right: # Leaf yield [node.value] else: yield from ([node.value] + arr for arr in allPaths(node.left)) yield from ([node.value] + arr for arr in allPaths(node.right)) root = Node(1) root.left = Node(2); root.left.left = Node(4); root.left.right = Node(5); root.right = Node(3); root.right.right = Node(6); g = allPaths(root) list(g) # [[1, 2, 4], [1, 2, 5], [1, 3, 6]]
Question: I'm trying to learn Python/Pygame and I made a simple Pong game. However I cannot get the square to bounce off the sides at the perfect pixel as the drawing of the square is updating let's say 3 pixels every frame. I have a code to decide when the square is hitting the edges and bounce in a reverse direction like this: if y_ball < 100: y_ball_change = y_ball_change * -1 if y_ball > 675: y_ball_change = y_ball_change * -1 if x_ball > 775: x_ball_change = x_ball_change * -1 if x_ball <= x_rect + 25 and y_ball >= y_rect -25 and not y_ball > y_rect + 150: x_ball_change = x_ball_change * -1 +2  It's keeping the square inside the boundaries of the screen however it's not pixel perfect since x_ball_change y_ball_change  is often more/less than 1 since they are randomized between -5 and 5 (except 0) to make starting direction of the ball random every new game. Thanks for any help!

Answer: You also need to correct the position of the ball when changing the direction of the ball. The ball bounces on the boundaries and moves the excessive distance in the opposite direction like a billiard ball: e.g.: if y_ball < 100: y_ball = 100 + (100 - y_ball) y_ball_change = y_ball_change * -1 if y_ball > 675: y_ball = 675 - (y_ball - 675) y_ball_change = y_ball_change * -1 if x_ball > 775: x_ball = 775 - (x_ball - 775) x_ball_change = x_ball_change * -1 if x_ball <= x_rect + 25 and y_rect -25 <= y_ball <= y_rect + 150: x_ball = x_rect + 25 + (x_rect + 25 - x_ball) x_ball_change = x_ball_change * -1 +2
Question: We have deployed a django server (nginx/gunicorn/django) but to scale the server there are multiple instances of same django application running. Here is the diagram (architecture):  Each blue rectangle is a Virtual Machine. HAProxy sends all request to example.com/admin to Server 3.other requests are divided between Server 1 and Server 2.(load balance). Old Problem: Each machine has a media folder and when admin Uploads something the uploaded media is only on Server 3. (normal users can't upload anything) We solved this by sending all requests to example.com/media/* to Server 3 and nginx from Server3 serves all static files and media. Problem right now We are also using sorl-thumbnail. When a requests comes for example.com/,sorl-thumbnail tries to access the media file but it doesn't exist on this machine because it's on Server3. So now all requests to that machine(server 1 or 2) get 404 for that media file. One solution that comes to mind is to make a shared partition between all 3 machines and use it as media. Another solution is to sync all media folders after each upload but this solution has problem and that is we have almost 2000 requests per second and sometimes sync might not be fast enough and sorl-thumbnail creates the database record of empty file and 404 happens. Thanks in advance and sorry for long question.

Answer: You should use an object store to save and serve your user uploaded files. django-storages makes the implementation really simple. If you don’t want to use cloud based AWS S3 or equivalent, you can host your own on-prem S3 compatible object store with minio. On your current setup I don’t see any easy way to fix where the number of vm s are dynamic depending on load. If you have deployment automation then maybe try out rsync so that the vm takes care of syncing files with other vms.
Question: I am trying to build a scalable method to calculate the number of unique members that have modified a certain file up to and including the latest modified_date. The unique_member_until_now column contains expected result for each file. import pandas as pd from pandas import Timestamp # Example Dataset df = pd.DataFrame({'File': ['A', 'A', 'A', 'B', 'B', 'B', 'B', 'B', 'B', 'C', 'C', 'C'], 'Member': ['X', 'X', 'Y', 'X', 'Y', 'Y', 'X', 'Z', 'Y', 'X', 'Y', 'X'], 'modified_date': [Timestamp('2021-11-25 00:00:00'), Timestamp('2021-11-28 00:00:00'), Timestamp('2021-12-14 00:00:00'), Timestamp('2021-10-17 00:00:00'), Timestamp('2021-11-01 00:00:00'), Timestamp('2021-11-04 00:00:00'), Timestamp('2021-11-16 00:00:00'), Timestamp('2021-12-16 00:00:00'), Timestamp('2021-12-29 00:00:00'), Timestamp('2021-10-30 00:00:00'), Timestamp('2021-11-23 00:00:00'), Timestamp('2021-12-17 00:00:00')], 'unique_member_until_now': [1, 1, 2, 1, 2, 2, 2, 3, 3, 1, 2, 2]})   df.groupby("File")["Member"].transform('nunique') ofcourse doesn't give the intended result The current approach is to iterate over every group and each record in the group, but I am sure that is grossly inefficient and slow when dealing with millions for rows.

Answer: An efficient method would be to compute the (non) duplicated on the File+Member columns, then groupby File and cumsum: (~df[['File', 'Member']].duplicated()).groupby(df['File']).cumsum()  Saving as column: df['unique_member_until_now'] = (~df[['File', 'Member']].duplicated()).groupby(df['File']).cumsum()  output:  File Member modified_date unique_member_until_now 0 A X 2021-11-25 1 1 A X 2021-11-28 1 2 A Y 2021-12-14 2 3 B X 2021-10-17 1 4 B Y 2021-11-01 2 5 B Y 2021-11-04 2 6 B X 2021-11-16 2 7 B Z 2021-12-16 3 8 B Y 2021-12-29 3 9 C X 2021-10-30 1 10 C Y 2021-11-23 2 11 C X 2021-12-17 2