eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: My Anaconda Navigator (v1.9.12) has been prompting me to upgrade to 1.10.0. Only problem is, when I click "yes" on the update prompt (which should close the navigator and update it), nothing happens. No problem, I thought. I ran  conda update anaconda-navigator  in the terminal. To no avail (and yes, I read the doc online and ran "conda deactivate" beforehand), same with  conda install anaconda-navigator=1.10  Both ran for a while, but the desktop navigator is still on the old version. One thing to note: the Looking for incompatible packages line was taking way too long (hours with no notable progress), so I ctrl-c'ed out. But I ran these commands again they managed to finish running. Now I'm out of ideas, would anyone know what I can do to go through with the update? Thanks a lot!

Answer: I am having completely the same issue (same Navigator version on macOS). I think I have spent several hours of all possible solution and nothing helped. The only solution that worked was to uninstall and install again. The environment setup remains the same so there is nothing to lose (but still it is strange thought) I was following the process from the answer from this question: How to uninstall Anaconda completely from macOS
Question: Q: How do I exclude a folder from pycharm's duplicate check? Minimal Example: Say my pycharm project folder structure looks like this: project/main.py project/.backup/main_copy.py  How do I tell pycharm not to warn me that main.py and main_copy.py contain duplicate code?

Answer: Try to mark .backup as excluded via right-click on it -> Mark Directory as.
Question: Five out of ten times Pytube will send me this error when attempting to run my small testing script. Here's the script: import pytube import urllib.request from pytube import YouTube yt = YouTube('https://www.youtube.com/watch?v=3NCyD3XoJgM') print('Youtube video title is: ' + yt.title + '! Downloading now!')  Here's what I get: Traceback (most recent call last): File "youtube.py", line 6, in <module> yt = YouTube('https://www.youtube.com/watch?v=3NCyD3XoJgM') File "C:\Users\test\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.8_qbz5n2kfra8p0\LocalCache\local-packages\Python38\site-packages\pytube\__main__.py", line 91, in __init__ self.prefetch() File "C:\Users\test\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.8_qbz5n2kfra8p0\LocalCache\local-packages\Python38\site-packages\pytube\__main__.py", line 183, in prefetch self.js_url = extract.js_url(self.watch_html) File "C:\Users\test\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.8_qbz5n2kfra8p0\LocalCache\local-packages\Python38\site-packages\pytube\extract.py", line 143, in js_url base_js = get_ytplayer_config(html)["assets"]["js"] KeyError: 'assets'  I am very confused. I attempted to reinstall Python plus pytube but I can't seem to remedy this issue. It's increasingly perplexing that the script works half of the time, but not the other half.

Answer: For now fixed 100% with this: https://github.com/nficano/pytube/pull/767#issuecomment-716184994 With anyone else getting this error or issue, run this command in a terminal or cmd: python -m pip install git+https://github.com/nficano/pytube An update to pytubeX that hasn't been released with the pip installation yet. The GitHub link is the current dev explaining the situation.
Question: The spatial package imported from Scipy can measure the Euclidean distance between specified points. Is it possible to return the same measurement by using the Delaunay package? Using the df below, the average distance between all points is measured grouped by Time. However, I'm hoping to use Delaunay triangulation to measure the average distance. import pandas as pd import numpy as np import matplotlib.pyplot as plt from scipy.spatial import Delaunay df = pd.DataFrame({ 'Time' : [1,1,1,1,2,2,2,2], 'A_X' : [5, 5, 6, 6, 4, 3, 3, 4], 'A_Y' : [5, 6, 6, 5, 5, 6, 5, 6], }) def make_points(x): return np.array(list(zip(x['A_X'], x['A_Y']))) points = df.groupby("Time").apply(make_points) for p in points: tri = Delaunay(p) ax.triplot(*p.T, tri.simplices)  Average distance between all points can be measured using below but I'm hoping to incorporate Delaunay.  avg_dist = (df.groupby(['Time']) .apply(lambda x: spatial.distance.pdist (np.array(list(zip(x['A_X'], x['A_Y'])))) .mean() if len(x) > 1 else 0) .reset_index() )  Intended Output:  Time 0 0 1 1.082842 1 2 1.082842 

Answer: You can try this function from itertools import combinations import numpy as np def edges_with_no_replacement(points): # get the unique coordinates points = np.unique(points.loc[:,['A_X','A_Y']].values,return_index=False,axis=0) if len(points) <= 1: return 0 # for two points, no triangle # I think return the distance between the two points make more sense? You can change the return value to zero. if len(points) == 2: return np.linalg.norm(points[0]-points[1]) tri = Delaunay(points) triangles = tri.simplices # get all the unique edges all_edges = set([tuple(sorted(edge)) for item in triangles for edge in combinations(item,2)]) # compute the average dist return np.mean([np.linalg.norm(points[edge[0]]-points[edge[1]]) for edge in all_edges])  This function will first find all the unique edges given triangles, then return the average length of the triangle edges. Apply this function avg_dist = (df.groupby(['Time']).apply(edges_with_no_replacement).reset_index())  The output is  Time 0 0 1 1.082843 1 2 1.082843   Note that the function edges_with_no_replacement will still throw QhullError if points are on the same line, for example Delaunay(np.array([[1,2],[1,3],[1,4]]))  So, you have to make sure the points are not on the same line.
Question: I have used poetry to create a wheel file. I am running following spark-submit command , but it is not working. I think I am missing something spark-submit --py-files /path/to/wheel  Please note that I have referred to below as well, but did not get much details as I am new to Python. how to pass python package to spark job and invoke main file from package with arguments

Answer: Wheel file can be executed as a part of below spark-submit command spark-submit --deploy-mode cluster --py-files /path/to/wheel main_file.py
Question: I have the following numpy array: import numpy as np pair_array = np.array([(205, 254), (205, 382), (254, 382), (18, 69), (205, 382), (31, 183), (31, 267), (31, 382), (183, 267), (183, 382)]) print(pair_array) #[[205 254] # [205 382] # [254 382] # [ 18 69] # [205 382] # [ 31 183] # [ 31 267] # [ 31 382] # [183 267] # [183 382]]  Is there a way to transform this array to a symmetric pandas Dataframe that contains the count of occurences for all possible combinations? I expect something along the lines of this: # 18 31 69 183 205 254 267 382 # 18 0 0 1 0 0 0 0 0 # 31 0 0 0 1 0 0 1 1 # 69 1 0 0 0 0 0 0 0 # 183 0 1 0 0 0 0 1 1 # 205 0 0 0 0 0 1 0 2 # 254 0 0 0 0 1 0 0 1 # 267 0 1 0 1 0 0 0 0 # 382 0 1 0 1 2 1 0 0 

Answer: One way could be to build a graph using NetworkX and obtain the adjacency matrix directly as a dataframe with nx.to_pandas_adjacency. To account for the co-occurrences of the edges in the graph, we can create a nx.MultiGraph, which allows for multiple edges connecting the same pair of nodes: import networkx as nx G = nx.from_edgelist(pair_array, create_using=nx.MultiGraph) nx.to_pandas_adjacency(G, nodelist=sorted(G.nodes()), dtype='int') 18 31 69 183 205 254 267 382 18 0 0 1 0 0 0 0 0 31 0 0 0 1 0 0 1 1 69 1 0 0 0 0 0 0 0 183 0 1 0 0 0 0 1 1 205 0 0 0 0 0 1 0 2 254 0 0 0 0 1 0 0 1 267 0 1 0 1 0 0 0 0 382 0 1 0 1 2 1 0 0  Building a NetworkX graph, will also enable to create an adjacency matrix or another depending on the behaviour we expect. We can either create it using a:  nx.Graph: If we want to set to 1 both entries (x,y) and (y,x) for a (x,y) (or (y,x)) edge. This will hence produce a symmetric adjacency matrix nx.DiGraph: If (x,y) should only set the (x,y) the entry to 1 nx.MultiGraph: For the same behaviour as a nx.Graph but accounting for edge co-occurrences nx.MultiDiGraph: For the same behaviour as a nx.DiGraph but also accounting for edge co-occurrences
Question: I am doing multi-class classification for a recommender system (item recommendations), and I'm currently training my network using sparse_categorical_crossentropy loss. Therefore, it is reasonable to perform EarlyStopping by monitoring my validation loss, val_loss as such: tf.keras.callbacks.EarlyStopping(monitor='val_loss', patience=10)  which works as expected. However, the performance of the network (recommender system) is measured by Average-Precision-at-10, and is tracked as a metric during training, as average_precision_at_k10. Because of this, I could also perform early stopping with this metric as such: tf.keras.callbacks.EarlyStopping(monitor='average_precision_at_k10', patience=10)  which also works as expected. My problem: Sometimes the validation loss increases, whilst the Average-Precision-at-10 is improving and vice-versa. Because of this, I would need to monitor both, and perform early stopping, if and only if both are deteriorating. What I would like to do: tf.keras.callbacks.EarlyStopping(monitor=['val_loss', 'average_precision_at_k10'], patience=10)  which obviously does not work. Any ideas how this could be done?

Answer: With guidance from Gerry P above I managed to create my own custom EarlyStopping callback, and thought I post it here in case anyone else are looking to implement something similar. If both the validation loss and the mean average precision at 10 does not improve for patience number of epochs, early stopping is performed. class CustomEarlyStopping(keras.callbacks.Callback): def __init__(self, patience=0): super(CustomEarlyStopping, self).__init__() self.patience = patience self.best_weights = None def on_train_begin(self, logs=None): # The number of epoch it has waited when loss is no longer minimum. self.wait = 0 # The epoch the training stops at. self.stopped_epoch = 0 # Initialize the best as infinity. self.best_v_loss = np.Inf self.best_map10 = 0 def on_epoch_end(self, epoch, logs=None): v_loss=logs.get('val_loss') map10=logs.get('val_average_precision_at_k10') # If BOTH the validation loss AND map10 does not improve for 'patience' epochs, stop training early. if np.less(v_loss, self.best_v_loss) and np.greater(map10, self.best_map10): self.best_v_loss = v_loss self.best_map10 = map10 self.wait = 0 # Record the best weights if current results is better (less). self.best_weights = self.model.get_weights() else: self.wait += 1 if self.wait >= self.patience: self.stopped_epoch = epoch self.model.stop_training = True print("Restoring model weights from the end of the best epoch.") self.model.set_weights(self.best_weights) def on_train_end(self, logs=None): if self.stopped_epoch > 0: print("Epoch %05d: early stopping" % (self.stopped_epoch + 1))  It is then used as: model.fit( x_train, y_train, batch_size=64, steps_per_epoch=5, epochs=30, verbose=0, callbacks=[CustomEarlyStopping(patience=10)], )
Question: I've a Python package package_name which provides a command line application command-line-app-name as console_script: setup.py: setup( ... entry_points={"console_scripts": ["command-line-app-name=package_name.cli:main"]}, ... )  The virtualenv is located in <project>/.venv and managed with pipenv. pipenv managed venvs should support VSCode debugging integration. I've created a debugger configuration launch.json file with setting the Python path to the venv (pythonPath): { // Use IntelliSense to learn about possible attributes. // Hover to view descriptions of existing attributes. // For more information, visit: https://go.microsoft.com/fwlink/?linkid=830387 "version": "0.2.0", "configurations": [ { "name": "Python: command-line-app-name", "type": "python", "request": "launch", "stopOnEntry": false, "program": "command-line-app-name", "linux": { "pythonPath": "${workspaceFolder}/.venv/bin/python", "args": ["-r", "/home/florian/gitlab/package_name/data/Test_MRM.d"] }, "windows": { "pythonPath": "${workspaceFolder}/.venv/Scripts/python.exe", "args": ["-r", "D:\\MassHunter\\Data\\demo_0000.d"], }, "console": "integratedTerminal" } ] }  The Windows and Linux specific venv python executable and command line arguments should not have an impact. If I run the debugger I get: FileNotFoundError: [Errno 2] No such file or directory: '/home/florian/gitlab/package-name/command-line-app-name'. It seems like I'm miss-interpreting the documentation somehow. I tried to find help w.r.t. vscode-python as well as debugpy without success. How can I debug a console script command line app (instead of a package module)?

Answer: console_scripts cannot be debugged out-of-the-box. The solution is to call the entry point function directly instead ("program": "${workspaceRoot}/package_name/cli.py",). This requires to add the if __name__ == '__main__': idiom in the corresponding module (here: cli.py). In my case the command line argument parser used is click. However the pseudo-code should be very similar for other command line parser libs. package_name/cli.py: @click.command() @click.option(...) def main(<args>, <kwargs>): ... if __name__ == '__main__': main() # pylint: disable=no-value-for-parameter  .vscode/launch.json: { // Use IntelliSense to learn about possible attributes. // Hover to view descriptions of existing attributes. // For more information, visit: https://go.microsoft.com/fwlink/?linkid=830387 "version": "0.2.0", "configurations": [ { "name": "Python: command-line-app-name", "type": "python", "request": "launch", "stopOnEntry": false, "program": "${workspaceRoot}/package_name/cli.py", "linux": { "pythonPath": "${workspaceFolder}/.venv/bin/python", "args": ["-r", "/home/florian/gitlab/package_name/data/Test_MRM.d"] }, "windows": { "pythonPath": "${workspaceFolder}/.venv/Scripts/python.exe", "args": ["-r", "D:\\MassHunter\\Data\\demo_0000.d"], }, "console": "integratedTerminal" } ] }  NOTE: The tool used to manage the venv makes a difference. This solution does work in case the venv is managed with pipenv. The solution does not work in case the venv is managed with poetry.
Question: I need to count the number of values in each row that are greater than zero and store them in a new column The df bellow:  team goals goals_against games_in_domestic_league 0 juventus 1 0 0 1 barcelona 0 1 1 2 santos 2 1 2  Should become:  team goals goals_against games_in_domestic_league total 0 juventus 1 0 0 1 1 barcelona 0 1 1 2 2 santos 2 1 2 3 

Answer: First idea is select numeric columns, test if greater like 0 and count Trues by sum: df['total'] = df.select_dtypes(np.number).gt(0).sum(axis=1)  If want specify columns by list: cols = ['goals','goals_against','games_in_domestic_league'] df['total'] = df[cols].gt(0).sum(axis=1)
Question: I'm developing a grid based game in pygame, and want the window to be resizable. I accomplish this with the following init code: pygame.display.set_mode((740, 440), pygame.RESIZABLE)  As well as the following in my event handler: elif event.type == pygame.VIDEORESIZE: game.screen = pygame.display.set_mode((event.w, event.h), pygame.RESIZABLE) # Code to re-size other important surfaces  The problem I'm having is that it seems a pygame.VIDEORESIZE event is only pushed once the user is done resizing the window, i.e. lets go of the border. screen.get_size() updates similarly. Since the graphics of my game are very simple, I'd really prefer for them to resize as the user drags the window. This is trivial in many other languages, but I can't find any reference for it in pygame - although I can't imagine a feature this basic would be impossible. How can I update my game as the screen is being resized in pygame? EDIT: Here is a minimal working example. Running on Windows 10, pygame 1.9.4, the following code will only draw the updated rectangle after the user finishes dragging the window. import sys import pygame pygame.init() size = 320, 240 black = 0, 0, 0 red = 255, 0, 0 screen = pygame.display.set_mode(size, pygame.RESIZABLE) while True: for event in pygame.event.get(): if event.type == pygame.QUIT: sys.exit() elif event.type == pygame.VIDEORESIZE: pygame.display.set_mode((event.w, event.h), pygame.RESIZABLE) screen.fill(black) pygame.draw.rect(screen, red, (10,10,screen.get_width(),screen.get_height())) pygame.display.flip() 

Answer: If you run into this kind of problem, it's always worth to google it using SDL instead of pygame, since pygame is a pretty low-level SDL wrapper. So that's not a problem of pygame itself, but rather how sdl and your window manager interact, e.g. see this SDL bug report. Nonetheless, if you really need to update the window while resizing, if you're using Windows, you can listen for the actual WM_SIZE event of Windows, redraw your screen, and update the "Windows"-window by calling RedrawWindow. Here's a simple example: import pygame import win32gui import win32con def wndProc(oldWndProc, draw_callback, hWnd, message, wParam, lParam): if message == win32con.WM_SIZE: draw_callback() win32gui.RedrawWindow(hWnd, None, None, win32con.RDW_INVALIDATE | win32con.RDW_ERASE) return win32gui.CallWindowProc(oldWndProc, hWnd, message, wParam, lParam) def main(): pygame.init() screen = pygame.display.set_mode((320, 240), pygame.RESIZABLE | pygame.DOUBLEBUF) def draw_game(): screen.fill(pygame.Color('black')) pygame.draw.rect(screen, pygame.Color('red'), pygame.Rect(0,0,screen.get_width(),screen.get_height()).inflate(-10, -10)) pygame.display.flip() oldWndProc = win32gui.SetWindowLong(win32gui.GetForegroundWindow(), win32con.GWL_WNDPROC, lambda *args: wndProc(oldWndProc, draw_game, *args)) while True: for event in pygame.event.get(): if event.type == pygame.QUIT: return elif event.type == pygame.VIDEORESIZE: pygame.display.set_mode((event.w, event.h), pygame.RESIZABLE| pygame.DOUBLEBUF) draw_game() if __name__ == '__main__': main()  Default behaviour:  With RedrawWindow:
Question: I'm developing some python microservices with grpc and i'm using docker for the cassandra database and the microservices. Is there a way to setup reload on change within docker-compose? I'm guessing that first I need the code mounted as a volume but I don't see a way to reload on GRPC server like for example flask does.

Answer: We use watchdog[watchmedo] with our grpc services and Docker. Install watchdog or add to your requirements.txt file python -m pip install watchdog[watchmedo] Then in your docker-compose.yml add watchmedo auto-restart --recursive --pattern="*.py" --directory="/usr/src/app/" python -- -m app to your container where --directory is the directory to where your app is contained inside the docker container, and python -- -m app is the file that starts your grpc Server. In this example the file that starts the server is called app.py:  app: build: context: ./app/ dockerfile: ./Dockerfile target: app command: watchmedo auto-restart --recursive --pattern="*.py" --directory="/usr/src/app/" python -- -m app volumes: - ./app/:/usr/src/app/
Question: >>> print (12//0.2) 59.0 >>> print(floor(12/0.2)) 60  Why floor division is not working according to the rule in this case? p.s. Here Python is treating 0.2 as 0.20000000001 in the floor division case So (12/0.2000000001) is resulting in 59.999999... And floor(59.999999999) outputting 59 But don't know why python is treating 0.2 as 0.2000000001in the floor division case but not in the division case?

Answer: The reason why 12 / 0.2 results in 60.0, is not because 0.2 is treated differently, but because the error in the floating point division cancels the error in the representation of 0.2. The float always has the same value (greater than decimal 0.2), but depending on the operations those errors will either accumulate or be cancelled. In other cases the error is not completely cancelled and shows up in the result: >>> (12 / (0.2 * 0.2)) * 0.2 59.99999999999999  In CPython integer division for these specific types (float // float after the first param is automatically converted) and relative magnitudes is performed as follows (see Python's source code for the full method): mod = a % b result = (a - mod) / b  If b was actually 0.2, then mod would be 0, but in floating point it is slightly larger, so mod is just under 0.2. If you do this manually you can see how we end up with 59.0: >>> a = 12.0 >>> b = 0.2 >>> mod = a % b >>> mod 0.19999999999999934 >>> (a - mod) / b 59.0  The OP is also asking about the error in the floating point division, here's that as well: The values (mantissa * base^exponent): 12: 1.1000000000000000000000000000000000000000000000000000 * 2^3 0.2: 1.1001100110011001100110011001100110011001100110011010 * 2^(-3)  Remember 0.2 is not really 0.2, it's 0.200000000000000011102230246251565404236316680908203125. The result of dividing 12 by a value that is > 0.2 should be < 60. To divide the values, we divide the mantissa and subtract the exponent, so we get: 12 / 0.2: 0.1110111111111111111111111111111111111111111111111111111 * 2^6  But the last 3 bits don't fit into a double, which only has 53 bits for the mantissa (including the sign) and we're currently using 56. Since the result starts with 0, we first normalise, multiplying the mantissa by 2 and subtracting one from the exponent. And then we have to round to the nearest 53 bit mantissa: normalised: 1.110111111111111111111111111111111111111111111111111111 * 2^5 rounded: 1.1110000000000000000000000000000000000000000000000000 * 2^5  1.1110000000000000000000000000000000000000000000000000 * 2^5 is equal to 60. The difference between the correct result (1.110111111111111111111111111111111111111111111111111111 * 2^5) and the closest value we can represent as a 64 bit double (1.1110000000000000000000000000000000000000000000000000 * 2^5) is the error in the floating point division.
Question: I'm trying to plot multiple arrows between two sets of scatter points. Plotting a line is easy enough with ax.plot. But I'm trying to implement an arrow instead of a line. The arrows don't appear to be aligning between the points. So if the line plot is initialised below, it works fine. But the quiver plot does not plot alone the same lines. import pandas as pd import matplotlib.pyplot as plt import numpy as np df = pd.DataFrame(np.random.randint(-50,50,size=(100, 4)), columns=list('ABCD')) fig, ax = plt.subplots() x1 = df['A'] y1 = df['B'] x2 = df['C'] y2 = df['D'] AB = plt.scatter(x1, y1, c = 'blue', marker = 'o', s = 10, zorder = 3) CD = plt.scatter(x2, y2, c = 'red', marker = 'o', s = 10, zorder = 2) # plot line between points #ax.plot([x1,x2],[y1,y2], color = 'black', linestyle = '--', linewidth = 0.5) ax.quiver([x1, x2], [y1, y2]) 

Answer: According to the documentation, see scale_units option, you need: angles='xy', scale_units='xy', scale=1 in quiver: AB = ax.scatter(x1, y1, c = 'blue', marker = 'o', s = 10, zorder = 3) CD = ax.scatter(x2, y2, c = 'red', marker = 'o', s = 10, zorder = 2) ax.quiver(x1, y1, (x2-x1), (y2-y1), angles='xy', scale_units='xy', scale=1) plt.show()  Output: