eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I am working on a project where pre-commit==2.15.0 was added to the python requirements file. I installed the requirements. Now when I try to do a git commit I get the following error: An unexpected error has occurred: ExecutableNotFoundError: Executable `/bin/sh` not found Check the log at C:\Users\username\.cache\pre-commit\pre-commit.log  In my pre-commit log I have: pre-commit version: 2.15.0 sys.version: 3.9.0 (tags/v3.9.0:9cf6752, Oct 5 2020, 15:34:40) [MSC v.1927 64 bit (AMD64)] sys.executable: c:\users\username\appdata\local\programs\python\python39\python.exe os.name: nt sys.platform: win32 Traceback (most recent call last): File "c:\users\username\appdata\local\programs\python\python39\lib\site-packages\pre_commit\error_handler.py", line 65, in error_handler yield File "c:\users\username\appdata\local\programs\python\python39\lib\site-packages\pre_commit\main.py", line 368, in main return hook_impl( File "c:\users\username\appdata\local\programs\python\python39\lib\site-packages\pre_commit\commands\hook_impl.py", line 231, in hook_impl retv, stdin = _run_legacy(hook_type, hook_dir, args) File "c:\users\username\appdata\local\programs\python\python39\lib\site-packages\pre_commit\commands\hook_impl.py", line 42, in _run_legacy cmd = normalize_cmd((legacy_hook, *args)) File "c:\users\username\appdata\local\programs\python\python39\lib\site-packages\pre_commit\parse_shebang.py", line 82, in normalize_cmd exe = normexe(cmd[0]) File "c:\users\username\appdata\local\programs\python\python39\lib\site-packages\pre_commit\parse_shebang.py", line 61, in normexe _error('not found') File "c:\users\username\appdata\local\programs\python\python39\lib\site-packages\pre_commit\parse_shebang.py", line 51, in _error raise ExecutableNotFoundError(f'Executable `{orig}` {msg}') pre_commit.parse_shebang.ExecutableNotFoundError: Executable `/bin/sh` not found  I work in Windows where my teammates work on Macs. It looks like precommit is trying to reference the /bin/sh script which is not in Windows. How do I get this precommit working?

Answer: your previous git hook is using a non-portable shebang (#!/bin/sh) (in your case this file will be located at .git/hooks/pre-commit.legacy -- originally .git/hooks/pre-commit) if you adjust the tool to use #!/usr/bin/env sh then pre-commit will be able to run it (even on windows) alternatively, if you don't want to use pre-commit in migration mode run pre-commit install --force you're also using an outdated version of pre-commit which may be contributing to your issues -- so I'd recommend upgrading that was well  disclaimer: I created pre-commit
Question: I just created twitter api with elevated access but I can't generate access token with write permission. I need this permission to update my twitter name. 

Answer: App settings -> User authentication Settings -> Edit. Enable OAuth 1.0A ->  App permissions Read and Write Callback URL can be set to something like http://localhost if you are not going to implement sign-in for other users Website URL set to something valid like your blog ideally, or use http://example.com if you must -> Save  App Keys & Tokens -> Authentication tokens Regenerate Access Token and Secret This will grant read and write permission to the app for the account that owns the app. It will not let any the app write to any other account.
Question:  File "C:\Users\Codertjay\PycharmProjects\Teems_App_Kid\teems_app_kid\__init__.py", line 5, in <module> from .celery import app as celery_app File "C:\Users\Codertjay\PycharmProjects\Teems_App_Kid\teems_app_kid\celery.py", line 3, in <module> from celery import Celery File "C:\Users\Codertjay\PycharmProjects\brownie\Teems_App_Kid\lib\site-packages\celery\five.py", line 306, in __getattr__ module = __import__(self._object_origins[name], None, None, [name]) File "C:\Users\Codertjay\PycharmProjects\brownie\Teems_App_Kid\lib\site-packages\celery\app\__init__.py", line 14, in <module> from celery import _state File "C:\Users\Codertjay\PycharmProjects\brownie\Teems_App_Kid\lib\site-packages\celery\_state.py", line 20, in <module> from celery.utils.threads import LocalStack File "C:\Users\Codertjay\PycharmProjects\brownie\Teems_App_Kid\lib\site-packages\celery\utils\__init__.py", line 20, in <module> from collections import Callable ImportError: cannot import name 'Callable' from 'collections' (C:\Users\Codertjay\AppData\Local\Programs\Python\Python310\lib\collections\__init__.py) 

Answer: The offending line has been removed from Celery nearly 6 years ago. You should update the celery package to a recent version.
Question: If you have tensor arrays of different lengths across several gpu ranks, the default all_gather method does not work as it requires the lengths to be same. For example, if you have: if gpu == 0: q = torch.tensor([1.5, 2.3], device=torch.device(gpu)) else: q = torch.tensor([5.3], device=torch.device(gpu))  If I need to gather these two tensor arrays as follows: all_q = [torch.tensor([1.5, 2.3], torch.tensor[5.3]) the default torch.all_gather does not work as the lengths, 2, 1 are different.

Answer: As it is not directly possible to gather using built in methods, we need to write custom function with the following steps:  Use dist.all_gather to get sizes of all arrays. Find the max size. Pad local array to max size using zeros/constants. Use dist.all_gather to get all padded arrays. Unpad the added zeros/constants using sizes found in step 1.  The below function does this: def all_gather(q, ws, device): """ Gathers tensor arrays of different lengths across multiple gpus Parameters ---------- q : tensor array ws : world size device : current gpu device Returns ------- all_q : list of gathered tensor arrays from all the gpus """ local_size = torch.tensor(q.size(), device=device) all_sizes = [torch.zeros_like(local_size) for _ in range(ws)] dist.all_gather(all_sizes, local_size) max_size = max(all_sizes) size_diff = max_size.item() - local_size.item() if size_diff: padding = torch.zeros(size_diff, device=device, dtype=q.dtype) q = torch.cat((q, padding)) all_qs_padded = [torch.zeros_like(q) for _ in range(ws)] dist.all_gather(all_qs_padded, q) all_qs = [] for q, size in zip(all_qs_padded, all_sizes): all_qs.append(q[:size]) return all_qs  Once, we are able to do the above, we can then easily use torch.cat to further concatenate into a single array if needed: torch.cat(all_q) [torch.tensor([1.5, 2.3, 5.3])  Adapted from: github
Question: Python lists have nifty indexing/slicing capabilities. Here are several examples: x = "123456" x[:-3] '123' x[:-1] '12345' # -1 slices off last element x[:-0] # -0 slices off .. everything .. this is what i'd like to fix ''  I would like to slice off a variable number of elements d: x[:-d]  But if d were 0 we get a much different result than desired. A workaround is: d = 0 x[:-d if d else len(x)] '123456'  That is possible - but is there any [shorter] alternative?

Answer: You could use None, which is the default for missing start/stop/step values: x[:-d or None]
Question: I have set up my service account and I can run queries on bigQuery using client.query(). I could just write all my scheduled queries into this new client.query() format but I already have many scheduled queries so I was wondering if there is a way I can get/list the scheduled queries and then use that information to run those queries from a script. 

Answer: Yes, you can use the APIs. When you don't know which one to use, I have a tip. Use the command proposed by @Yev bq ls --transfer_config --transfer_location=US --format=prettyjson But log the API calls. for that use the --apilog <logfile name> parameter like that bq --apilog ./log ls --transfer_config --transfer_location=US --format=prettyjson And, magically, you can find the API called by the command: https://bigquerydatatransfer.googleapis.com/v1/projects/<PROJECT-ID>/locations/US/transferConfigs?alt=json Then, a simple google search leads you to the correct documentation  In python, add that dependencies in your requirements.txt: google-cloud-bigquery-datatransfer and use that code from google.cloud import bigquery_datatransfer client = bigquery_datatransfer.DataTransferServiceClient() parent = client.common_project_path("<PROJECT-ID>") resp = client.list_transfer_configs(parent=parent) print(resp)
Question: I have the following code for plotting the histogram and the kde-functions (Kernel density estimation) of a training and validation dataset: #Plot histograms import matplotlib.pyplot as plt import matplotlib import seaborn as sns displot_dataTrain=sns.displot(data_train, bins='auto', kde=True) displot_dataTrain._legend.remove() plt.ylabel('Count') plt.xlabel('Training Data') plt.title("Histogram Training Data") plt.show() displot_dataValid =sns.displot(data_valid, bins='auto', kde=True) displot_dataValid._legend.remove() plt.ylabel('Count') plt.xlabel('Validation Data') plt.title("Histogram Validation Data") plt.show() # Try to plot the kde-functions together --> yields an AttributeError X1 = np.linspace(data_train.min(), data_train.max(), 1000) X2 = np.linspace(data_valid.min(), data_valid.max(), 1000) fig, ax = plt.subplots(1,2, figsize=(12,6)) ax[0].plot(X1, displot_dataTest.kde.pdf(X1), label='train') ax[1].plot(X2, displot_dataValid.kde.pdf(X1), label='valid')  The plotting of the histograms and kde-functions inside one plot works without problems. Now I would like to have the 2 kde-functions inside one plot but when using the posted code, I get the following error AttributeError: 'FacetGrid' object has no attribute 'kde' Do you have any idea, how I can combined the 2 kde-functions inside one plot (without the histogram)?

Answer: sns.displot() returns a FacetGrid. That doesn't work as input for ax.plot(). Also, displot_dataTest.kde.pdf is never valid. However, you can write sns.kdeplot(data=data_train, ax=ax[0]) to create a kdeplot inside the first subplot. See the docs; note the optional parameters cut= and clip= that can be used to adjust the limits. If you only want one subplot, you can use fig, ax = plt.subplots(1, 1, figsize=(12,6)) and use ax=ax instead of ax=ax[0] as in that case ax is just a single subplot, not an array of subplots. The following code has been tested using the latest seaborn version: import matplotlib.pyplot as plt import seaborn as sns import numpy as np fig, ax = plt.subplots(figsize=(12, 6)) sns.kdeplot(data=np.random.normal(0.1, 1, 100).cumsum(), color='crimson', label='train', fill=True, ax=ax) sns.kdeplot(data=np.random.normal(0.1, 1, 100).cumsum(), color='limegreen', label='valid', fill=True, ax=ax) ax.legend() plt.tight_layout() plt.show()
Question: I'm trying to display an image on the screen, without any window/application popping up/containing it. I'm pretty close with TKinter, but the method for removing the background color of the canvas is hacky and has some undesired effects. import tkinter as tk import ctypes user32 = ctypes.windll.user32 screen_size = user32.GetSystemMetrics(0), user32.GetSystemMetrics(1) root = tk.Tk() root.overrideredirect(True) root.config(bg="blue", bd=0, highlightthickness=0) root.attributes("-transparentcolor", "#FEFCFD") root.attributes("-topmost", True) tk_img = tk.PhotoImage(file="image.png") canvas = tk.Canvas(root, bg="#FEFCFD", bd=0, highlightthickness=0, width=screen_size[0], height=screen_size[1]) canvas.pack() img = canvas.create_image(0, 0, image=tk_img, anchor="nw") root.mainloop()  The -transparentcolor flag mostly removes the background, but if an image has any partially transparent pixels it will tint them. Plus, if that color exists in the image, it will be removed; that choice of color was in hopes of minimizing exact matches in an image while also being mostly white, to hopefully have the least noticeable affect on the images. Here's an image of what it looks like currently; very close to what I want, but you can see some missing pixels in the white areas of the dice, and they all seem to have a white border around them due to their edges being partially transparent. This is what the image should look like. I've also tried to achieve this effect using wxPython, but I can't remove the background of the window, leading to transparent images always being backed by some color. I used this answer; I've modified it slightly but nothing I've done has improved it. So, is there a way to draw an image on the screen without any background at all with Python?

Answer: Thanks to the suggestion from Kartikeya, I was able to solve my own question. Using PyQt5, this code will display an image with transparency and no border or background at all import sys from PyQt5.QtCore import Qt from PyQt5.QtGui import QPixmap from PyQt5.QtWidgets import QMainWindow, QApplication, QLabel app = QApplication(sys.argv) window = QMainWindow() window.setAttribute(Qt.WA_TranslucentBackground, True) window.setAttribute(Qt.WA_NoSystemBackground, True) window.setWindowFlags(Qt.FramelessWindowHint) label = QLabel(window) pixmap = QPixmap('image.png') label.setPixmap(pixmap) label.setGeometry(0, 0, pixmap.width(), pixmap.height()) window.label = label window.resize(pixmap.width(),pixmap.height()) window.show() sys.exit(app.exec_())  Once I was looking for PyQt5, I found this question and only needed to modify the code slightly. Here is what it looks like now.
Question: I recently swicthed on the preview feature "files in repos" on Azure Databricks, so that I could move a lot of my general functions from notebooks to modules and get rid of the overhead from running a lot of notebooks for a single job. However, several of my functions rely directly on dbutils or spark/pyspark functions (e.g. dbutils.secrets.get() and spark.conf.set()). Since these are imported in the background for the notebooks and are tied directly to the underlying session, I am at complete loss as to how I can reference these modules in my custom modules. For my small sample module, I fixed it by making dbutils a parameter, like in the following example: class Connection: def __init__(self, dbutils): token = dbutils.secrets.get(scope="my-scope", key="mykey") ...  However, doing this for all the existing functions would require a significant amount of rewriting both the functions and the lines that call them. How can I avoid this procedure and do it in a more clean manner?

Answer: The documentation for Databricks Connect shows the example how it could be achieved. That example has SparkSession as an explicit parameter, but it could be modified to avoid that completely, with something like this: def get_dbutils(): from pyspark.sql import SparkSession spark = SparkSession.getActiveSession() if spark.conf.get("spark.databricks.service.client.enabled") == "true": from pyspark.dbutils import DBUtils return DBUtils(spark) else: import IPython return IPython.get_ipython().user_ns["dbutils"]  and then in your function you can do something like this: def get_my_secret(scope, key): return get_dbutils().secrets.get(scope, key)
Question: I would like to avoid for loops in this code snippet: import numpy as np N = 4 a = np.random.randint(0, 256, size=(N, N, 3)) m = np.random.randint(0, 2, size=(N, N)) for i, d0 in enumerate(a): for j, d1 in enumerate(d0): if m[i, j]: d1[2] = 42  This is a simplified example where a is an N x N RGB image and m is a N x N mask, which sets masked elements of the 3rd channel: a[:, :, 2] only.

Answer: You can index the last axis and set the elements selected by a boolean mask import numpy as np N = 4 a = np.random.randint(0, 256, size=(N, N, 3)) m = np.random.randint(0, 2, size=(N, N)) a[...,2][m.astype('bool')] = 42 a  Output (for a random example of a) array([[[ 9, 13, 4], [15, 0, 42], [11, 12, 9], [13, 0, 42]], [[ 1, 10, 42], [ 9, 0, 42], [ 8, 6, 4], [ 3, 0, 42]], [[15, 11, 6], [ 8, 11, 42], [14, 1, 42], [ 4, 14, 1]], [[ 3, 6, 42], [ 5, 13, 3], [ 9, 14, 13], [12, 6, 42]]])
Question: I use translation in my Django apps. Since I installed Django version 4, when I try to import ugettexget_lazy as shown in the code below from django.utils.translation import ugettexget_lazy as _  I get the following error: ImportError: cannot import name 'ugettext_lazy' from 'django.utils.translation' 

Answer: It was removed from Django 4, use this instead from django.utils.translation import gettext_lazy as _
Question: I am making a program that simulates a stock market for a virtual currency. I have not tried anything yet, but I want a Python script to run 24/7 online for a long period of time. The python script should be something like this: import time import random while True: number = random.randint(0,5) print(number) time.sleep(2)  Also, a separate local Python program should be able to retrieve the number variable constantly every 2 seconds. I do not need a recommendation for a product or service. I just need to know what code I need to run and if I need a physical or web server. If I use a web server, I will be paying monthly for a long time. This project needs to be running online for theoretically years (setting aside downtime and maintenance). I have barely any experience in servers and networking, and I couldn't find answers online. Again, I have not tried anything yet.

Answer: If you really want to just simulate it and there is only one user. You can just return a random number each time the user make a query. It’s irrelevant how often they query it. You can put this up locally or on heroic free plan. But the fact that the user is querying every 2 seconds means lots of requests and so you may exceed the quota. import random from flask import Flask import time app = Flask(__name__) @app.route("/") def hello_world(): return random.randint(0,5)  Say you up it up locally on port 5000. Then, simply going to “localhost:5000” via python or browser will give you that random number. To serve it up to multiple people, you want to have a separate thread running to generate the number. Then, the view would return the number at the URL. For example, from flask import Flask import random import threading app = Flask(__name__) number = 0 @app.route("/") def hello_world(): """ URL returns random number """ return str(number) def gen_rand(): """ generating random numbers """ global number while True: number = random.randint(0, 5) time.sleep(2) if __name__ == '__main__': # starting thread to generate the number x = threading.Thread(target=gen_rand, daemon=True) x.start() # starting web server app.run()  By default, the local webserver will start at localhost:5000, go to this URL in your browser and you will see the randomly generated numbered. You can open multiple browser tabs to see they will give same random number if refreshed within 2 seconds. Note that using global variable is not thread-safe or process safe. You should use database or redis to update and load “number”. See this question for further discussions: Are global variables thread-safe in Flask? How do I share data between requests?.
Question: I'm trying to pip install geopandas as a fresh installation, so I want to remove existing packages like GDAL and fiona. I've already managed to pip uninstall fiona, but when I try to uninstall or reinstall GDAL it gives the following error message: (base) C:\usr>pip install C:/usr/Anaconda3/Lib/site-packages/GDAL-3.4.1-cp38-cp38-win_amd64.whl Processing c:\usr\anaconda3\lib\site-packages\gdal-3.4.1-cp38-cp38-win_amd64.whl Installing collected packages: GDAL Attempting uninstall: GDAL Found existing installation: GDAL 3.0.2 ERROR: Exception: Traceback (most recent call last): File "C:\usr\Anaconda3\lib\site-packages\pip\_internal\cli\base_command.py", line 167, in exc_logging_wrapper status = run_func(*args) File "C:\usr\Anaconda3\lib\site-packages\pip\_internal\cli\req_command.py", line 205, in wrapper return func(self, options, args) File "C:\usr\Anaconda3\lib\site-packages\pip\_internal\commands\install.py", line 405, in run installed = install_given_reqs( File "C:\usr\Anaconda3\lib\site-packages\pip\_internal\req\__init__.py", line 68, in install_given_reqs uninstalled_pathset = requirement.uninstall(auto_confirm=True) File "C:\usr\Anaconda3\lib\site-packages\pip\_internal\req\req_install.py", line 637, in uninstall uninstalled_pathset = UninstallPathSet.from_dist(dist) File "C:\usr\Anaconda3\lib\site-packages\pip\_internal\req\req_uninstall.py", line 554, in from_dist for script in dist.iterdir("scripts"): File "C:\usr\Anaconda3\lib\site-packages\pip\_internal\metadata\pkg_resources.py", line 156, in iterdir if not self._dist.isdir(name): File "C:\usr\Anaconda3\lib\site-packages\pip\_vendor\pkg_resources\__init__.py", line 2816, in __getattr__ return getattr(self._provider, attr) AttributeError: 'PathMetadata' object has no attribute 'isdir'  Does anyone know why GDAL cannot be uninstalled?

Answer: I just came across this question after getting the same error. Coincidentally I had just upgraded pip (I was getting tired of the yellow warnings). All I had was to down grade my pip pip install pip==21.3.1 --user
Question: Why does this happen in Python?  float('inf') == float('inf') returns True, float('inf') + float('inf') == float('inf') returns True, float('inf') * float('inf') == float('inf') returns True, float('inf') - float('inf') == float('inf') returns False, float('inf') / float('inf') == float('inf') returns False.  My best guess, if I think about limits, is related with the result of this operation: limx→+∞(f(x) ▢ g(x)) where limx→+∞ f(x) = +∞ and limx→+∞ g(x) = +∞, which returns +∞ if ▢ is + or *, but it is not defined (it could return every value) if ▢ is - or /. I am very puzzled, though.

Answer: Before the comparison of float('inf') - float('inf') == float('inf')  can be made, the result of float('inf') - float('inf')  will be calculated. That result is NaN. It is NaN because the amounts of infinity may differ. It's explained on the Stack Overflow sister site Math.SE, known as the Hilbert hotel paradox:  From a layman's perspective, imagine that I have an infinite number of hotel rooms, each numbered 1, 2, 3, 4, ... Then I give you all of them. I would have none left, so ∞−∞=0 On the other hand, if I give you all of the odd-numbered ones, then I still have an infinite number left. So ∞−∞=∞. Now suppose that I give you all of them except for the first seven. Then ∞−∞=7. While this doesn't explain why this is indeterminate, hopefully you can agree that it is indeterminate!  The best number to represent indeterminate is NaN. Comparing NaN to anything is always False, even comparing NaN against itself. Besides that quite "logical" explanation, we find that Python uses IEEE754 representation for floating point calculation. You'd typically need to buy the IEEE754 specification, but luckily we see some draft version online. The relevant chapter IMHO is 7.2:  For operations producing results in floating-point format, the default result of an operation that signals the invalid operation exception shall be a quiet NaN [...] [...] d) addition or subtraction or fusedMultiplyAdd: magnitude subtraction of infinities, such as: addition(+∞, −∞)