eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I am trying to open a private google sheet using python. The end goal here is to read that private sheet data into a json object. I have made sure to create a google cloud project, enable the API's, and service account. The service account email has been shared and added as an editor. I also created OAuth keys for a desktop application. This is required since the file is private. I know I need to somehow request a token to use for access to the sheets API, but I am at a loss for how to create a request, and utilize the client_secret file generated from OAuth keys. I figured the googleAPI would have a function where you can pass this file directly, but I am lost in documentation. Any insight would be appreciated!

Answer: All you need to do is supply the library with the location of the clientSecret.json file you should have downloaded from Google cloud console. This method should build the service for you and you can make the requests to the api. It will handle all the authorization. from apiclient.discovery import build from oauth2client.service_account import ServiceAccountCredentials def get_service(api_name, api_version, scopes, key_file_location): """Get a service that communicates to a Google API. Args: api_name: The name of the api to connect to. api_version: The api version to connect to. scopes: A list auth scopes to authorize for the application. key_file_location: The path to a valid service account JSON key file. Returns: A service that is connected to the specified API. """ credentials = ServiceAccountCredentials.from_json_keyfile_name( key_file_location, scopes=scopes) # Build the service object. service = build(api_name, api_version, credentials=credentials) return service  The best example I know of for service account authentication with python is the Google analytics quickstart If you have any issues altering it for google sheets let me know i can try and help. Calling it should be something like this. def main(): # Define the auth scopes to request. scope = 'https://www.googleapis.com/auth/spreadsheets' key_file_location = '<REPLACE_WITH_JSON_FILE>' # Authenticate and construct service. service = get_service( api_name='sheets', api_version='v4', scopes=[scope], key_file_location=key_file_location) data = your_method_to_call_sheets(service)  How to create clientSecret.json remember to enable the google sheets api under libary
Question: Is there a generic way to convert all float64 values in a pandas dataframe to float32 values? But not changing uint16 to float32? I don't know the signal names in advance but just want to have no float64. Something like: if float64, then convert to float32, else nothing?  The structure of the data is: DF.dtypes Counter uint16 p_007 float64 p_006 float64 p_005 float64 p_004 float64 

Answer: Try this: df[df.select_dtypes(np.float64).columns] = df.select_dtypes(np.float64).astype(np.float32)
Question: Issue description: Getting AttributeError: module 'base64' has no attribute 'decodestring' error while running on python 3.9.6 Steps to reproduce: Below is a dummy program, while running on python 3.9.6, I am getting `AttributeError: module 'base64' has no attribute 'decodestring'`` error: from ldif3 import LDIFParser parser = LDIFParser(open('dse3.ldif', 'rb')) for dn, entry in parser.parse(): if dn == "cn=Schema Compatibility,cn=plugins,cn=config": if entry['nsslapd-pluginEnabled'] == ['on']: print('Entry record: %s' % dn)  Error message: python3.9 1.py  ✔    venvpy3.9   11:12:01  Traceback (most recent call last): File "/Users/rasrivas/local_test/1.py", line 4, in <module> for dn, entry in parser.parse(): File "/Users/rasrivas/local_test/venvpy3.9/lib/python3.9/site-packages/ldif3.py", line 384, in parse yield self._parse_entry_record(block) File "/Users/rasrivas/local_test/venvpy3.9/lib/python3.9/site-packages/ldif3.py", line 357, in _parse_entry_record attr_type, attr_value = self._parse_attr(line) File "/Users/rasrivas/local_test/venvpy3.9/lib/python3.9/site-packages/ldif3.py", line 315, in _parse_attr attr_value = base64.decodestring(line[colon_pos + 2:]) AttributeError: module 'base64' has no attribute 'decodestring'  python version python --version Python 3.9.6  Operating system: macOS 11.5.2 Python version: python --version Python 3.9.6  python-ldap version: ldif3-3.2.2

Answer: From the docs for Python 3.8, base64.decodestring() is described as a:  Deprecated alias of decodebytes().  It looks like the base64.decodestring() function has been deprecated since Python 3.1, and removed in Python 3.9. You will want to use the bas64.decodebytes() function instead.
Question:  balance = 100 print('Current Balance: ', balance) while balance > 0: print('1. WITHDRAW') print('2. DEPOSIT') choice = input("Select an option... ") if (choice == 1): print('1') elif (choice == 2): print('2') else: print('test')  When I run the code with the code runner extension the code shows in the terminal however when it gets to the input function it freezes like its asking me to input some data however I cant eve type a number or letter. This is what the terminal shows... [Running] python -u "c:\Users\bowen\Desktop\CSE 120\PROJECT 3\main.py" Current Balance: 100 1. WITHDRAW 2. DEPOSIT Select an option... 

Answer: Code Runner shows results in OUTPUT and doesn't accept inputs by default. Add "code-runner.runInTerminal": true  in Settings.json, then you can input data.
Question: I have a large set of 2D points that I've downsampled into a 44x2 numpy array (array defined later). I am trying to find the bounding shape of those points which are effectively a concave hull. In the 2nd image I've manually marked an approximate bounding shape that I am hoping to get.   I have tried using alphashape and the Delauney triangulation method from here, both methods providing the same answer. Unfortunately, I don't seem to be able to achieve what I need, regardless of the alpha parameters. I've tried some manual settings and alphaoptimize, some examples of which are below.    Is there something critical I'm misunderstanding about alphashape? The documentation seems very clear, but obviously I'm missing something. import numpy as np import alphashape from descartes import PolygonPatch import matplotlib.pyplot as plt points = np.array( [[0.16,3.98], [-0.48,3.33], [-0.48,4.53], [0.1,3.67], [0.04,5.67], [-7.94,3.02], [-18.16,3.07], [-0.15,5.67], [-0.26,5.14], [-0.1,5.11], [-0.96,5.48], [-0.03,3.86], [-0.12,3.16], [0.32,4.64], [-0.1,4.32], [-0.84,4.28], [-0.56,3.16], [-6.85,3.28], [-0.7,3.24], [-7.2,3.03], [-1.0,3.28], [-1.1,3.28], [-2.4,3.28], [-2.6,3.28], [-2.9,3.28], [-4.5,3.28], [-12.3,3.28], [-14.8,3.28], [-16.7,3.28], [-17.8,3.28], [-0,3.03], [-1,3.03], [-2.1,3.03], [-2.8,3.03], [-3.2,3.03], [-5,3.03], [-12,3.03], [-14,3.03], [-17,3.03], [-18,3.03], [-0.68,4.86], [-1.26,3.66], [-1.71,3.51], [-9.49,3.25]]) alpha = 0.1 alphashape = alphashape.alphashape(points, alpha) fig = plt.figure() ax = plt.gca() ax.scatter(points[:,0],points[:,1]) ax.add_patch(PolygonPatch(alphashape,alpha=0.2)) plt.show() 

Answer: The plots that you attached are misleading, since the scales on the x-axis and the y-axis are very different. If you set both axes to the same scale, you obtain the following plot: . Since differences between x-coordinates of points are on the average much larger than differences between y-coordinates, you cannot obtain an alpha shape resembling your desired result. For larger values of alpha points scattered along the x-axis will not be connected by edges, since alpha shape will use circles too small to connect these points. For values of alpha small enough that these points get connected you will obtain the long edges on the right-hand side of the plot. You can fix this issue by rescaling y-coordinates of all points, effectively stretching the plot in the vertical direction. For example, multiplying y-coordinates by 7 and setting alpha = 0.4 gives the following picture:
Question: I am trying to automate the scraping of a site with "infinite scroll" with Python and Playwright. The issue is that Playwright doesn't include, as of yet, a scroll functionnality let alone an infinite auto-scroll functionnality. From what I found on the net and my personnal testing, I can automate an infinite or finite scroll using the page.evaluate() function and some Javascript code. For example, this works: for i in range(20): page.evaluate('var div = document.getElementsByClassName("comment-container")[0];div.scrollTop = div.scrollHeight') page.wait_for_timeout(500)  The problem with this approach is that it will either work by specifying a number of scrolls or by telling it to keep going forever with a while True loop. I need to find a way to tell it to keep scrolling until the final content loads. This is the Javascript that I am currently trying in page.evaluate(): var intervalID = setInterval(function() { var scrollingElement = (document.scrollingElement || document.body); scrollingElement.scrollTop = scrollingElement.scrollHeight; console.log('fail') }, 1000); var anotherID = setInterval(function() { if ((window.innerHeight + window.scrollY) >= document.body.offsetHeight) { clearInterval(intervalID); }}, 1000)  This does not work either in my firefox browser or in the Playwright firefox browser. It returns immediately and doesn't execute the code in intervals. I would be grateful if someone could tell me how I can, using Playwright, create an auto-scroll function that will detect and stop when it reaches the bottom of a dynamically loading webpage.

Answer: The new Playwright version has a scroll function. it's called mouse.wheel(x, y). In the below code, we'll be attempting to scroll through youtube.com which has an "infinite scroll": from playwright.sync_api import Playwright, sync_playwright import time def run(playwright: Playwright) -> None: browser = playwright.chromium.launch(headless=False) context = browser.new_context() # Open new page page = context.new_page() page.goto('https://www.youtube.com/') # page.mouse.wheel(horizontally, vertically(positive is # scrolling down, negative is scrolling up) for i in range(5): #make the range as long as needed page.mouse.wheel(0, 15000) time.sleep(2) time.sleep(15) # --------------------- context.close() browser.close() with sync_playwright() as playwright: run(playwright)
Question: import cv2 import numpy as np import math import sys import matplotlib.pyplot as plt import utils as ut imgGray = cv2.imread(imgfile, cv2.IMREAD_GRAYSCALE) plt.imshow(imgGray, cmap = 'gray') plt.show() cv2.imshow("",imgGray) cv2.waitKey(0) cv2.destroyAllWindows() sys.exit()  plt.show() result  cv2.imshow() result  I thought both of them would be same. But as you can see, two pictures have different grayscale. Seems plt.show() darker than cv2.imshow() How do I have to make grayscale in plt.show() same as cv2.imshow()? Python : 3.9.6 opencv-python : 4.5.3.56 mathplotlib : 3.4.3

Answer: This is the behavior of matplotlib. It finds the minimum and maximum of your picture, makes those black and white, and scales everything in between. This is useful for arbitrary data that may have integer or floating point types, and value ranges between 0.0 and 1.0, or 0 .. 255, or anything else. You can set those limits yourself with vmin and vmax arguments: plt.imshow(imgGray, cmap='gray', vmin=0, vmax=255) # if your data ranges is uint8  OpenCV does no such auto-scaling. It has fixed rules. If it's floating point, 0.0 is black and 1.0 is white. If it's uint8, the range is 0 .. 255. To get such auto-ranging in OpenCV, you'll have to scale the data before displaying: normalized = cv.normalize( data, alpha=0.0, beta=1.0, norm_type=cv.NORM_MINMAX, dtype=cv.CV_32F)
Question: I am currently working on a huge project, which constantly executes queries. My problem is, that my old code always created a new database connection and cursor, which decreased the speed immensivly. So I thought it's time to make a new database class, which looks like this at the moment: class Database(object): _instance = None def __new__(cls): if cls._instance is None: cls._instance = object.__new__(cls) try: connection = Database._instance.connection = mysql.connector.connect(host="127.0.0.1", user="root", password="", database="db_test") cursor = Database._instance.cursor = connection.cursor() except Exception as error: print("Error: Connection not established {}".format(error)) else: print("Connection established") return cls._instance def __init__(self): self.connection = self._instance.connection self.cursor = self._instance.cursor # Do database stuff here  The queries will use the class like so: def foo(): with Database() as cursor: cursor.execute("STATEMENT")  I am not absolutly sure, if this creates the connection only once regardless of how often the class is created. Maybe someone knows how to initialize a connection only once and how to make use of it in the class afterwards or maybe knows if my solution is correct. I am thankful for any help!

Answer: Explanation The keyword here is clearly class variables. Taking a look in the official documentation, we can see that class variables, other than instance variables, are shared by all class instances regardless of how many class instances exists.  Generally speaking, instance variables are for data unique to each instance and class variables are for attributes and methods shared by all instances of the class:  So let us asume you have multiple instances of the class. The class itself is defined like below. class Dog: kind = "canine" # class variable shared by all instances def __init__(self, name): self.name = name # instance variable unique to each instance  In order to better understand the differences between class variables and instance variables, I would like to include a small example here: >>> d = Dog("Fido") >>> e = Dog("Buddy") >>> d.kind # shared by all dogs "canine" >>> e.kind # shared by all dogs "canine" >>> d.name # unique to d "Fido" >>> e.name # unique to e "Buddy"  Solution Now that we know that class variables are shared by all instances of the class, we can simply define the connection and cursor like shown below. class Database(object): connection = None def __init__(self): if Database.connection is None: try: Database.connection = mysql.connector.connect(host="127.0.0.1", user="root", password="", database="db_test") except Exception as error: print("Error: Connection not established {}".format(error)) else: print("Connection established") def execute_query(self, sql): cursor = Database.connection.cursor() cursor.execute(sql)  As a result, the connection to the database is created once at the beginning and can then be used by every further instance. Note that the cursor is not cached, since it takes essentially no time at all to create a cursor. However, creating a connection is quite expensive, so it is sensible to cache them.
Question: I'm using the flake8 linter for Python and I have many code formats issues like blank line contains whitespace flake8(W293) I'm trying to auto fix these linting issues. I have these settings:  "python.linting.enabled": true, "python.linting.flake8Enabled": true, "python.linting.lintOnSave": true, "python.linting.flake8Args": [ "--ignore=E501", ], "editor.formatOnSave": true  I'm using the default autopep8 formater but it seems that it does nothing. Nothing happens when I save the file or run the command Format Document. Is there any way to auto fix these linting errors?

Answer: I would suggest using a formatter, black for instance, to fix the issues detected by your linter. If so, pip install it and add this to your settings.json: "python.formatting.provider": "black"  Then, pressing Alt+ShifT+F or Ctrl+S should trigger the formatting of your script.
Question: Say the two series are: x = [4,4,4,4,6,8,10,8,6,4,4,4,4,4,4,4,4,4,4,4,4,4,4] y = [4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,6,8,10,8,6,4,4]  Series x clearly lags y by 12 time periods. However, using the following code as suggested in Python cross correlation: import numpy as np c = np.correlate(x, y, "full") lag = np.argmax(c) - c.size/2  leads to an incorrect lag of -0.5. What's wrong here?

Answer: If you want to do it the easy way you should simply use scipy correlation_lags Also, remember to subtract the mean from the inputs. import numpy as np from scipy import signal x = [4,4,4,4,6,8,10,8,6,4,4,4,4,4,4,4,4,4,4,4,4,4,4] y = [4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,6,8,10,8,6,4,4] correlation = signal.correlate(x-np.mean(x), y - np.mean(y), mode="full") lags = signal.correlation_lags(len(x), len(y), mode="full") lag = lags[np.argmax(abs(correlation))]  This gives lag=-12, that is the difference between the index of the first six in x and in y, if you swap inputs it gives +12 Edit Why to subtract the mean If the signals have non-zero mean the terms at the center of the correlation will become larger, because there you have a larger support sample to compute the correlation. Furthermore, for very large data, subtracting the mean makes the calculations more accurate. Here I illustrate what would happen if the mean was not subtracted for this example. plt.plot(abs(correlation)) plt.plot(abs(signal.correlate(x, y, mode="full"))) plt.plot(abs(signal.correlate(np.ones_like(x)*np.mean(x), np.ones_like(y)*np.mean(y)))) plt.legend(['subtracting mean', 'constant signal', 'keeping the mean'])   Notice that the maximum on the blue curve (at 10) does not coincide with the maximum of the orange curve.
Question: How can I add the rewards to tensorboard logging in Stable Baselines3 using a custom environment? I have this learning code model = PPO( "MlpPolicy", env, learning_rate=1e-4, policy_kwargs=policy_kwargs, verbose=1, tensorboard_log="./tensorboard/") 

Answer: You can access the local variables available to the logger callback using self.locals. Any variables exposed in your custom environment will be accessible via locals dict. The example below shows how to access a key in a custom dictionary called my_custom_info_dict in vectorized environments. import numpy as np from stable_baselines3 import SAC from stable_baselines3.common.callbacks import BaseCallback from stable_baselines3.common.logger import TensorBoardOutputFormat from stable_baselines3.common.vec_env import SubprocVecEnv def make_env(env): """ See https://colab.research.google.com/github/Stable-Baselines-Team/rl-colab-notebooks/blob/sb3/multiprocessing_rl.ipynb for more details on vectorized environments Utility function for multiprocessed env. :param env_id: (str) the environment ID :param num_env: (int) the number of environment you wish to have in subprocesses :param seed: (int) the inital seed for RNG :param rank: (int) index of the subprocess :return: (Callable) """ def _init(): return env return _init class SummaryWriterCallback(BaseCallback): ''' Snippet skeleton from Stable baselines3 documentation here: https://stable-baselines3.readthedocs.io/en/master/guide/tensorboard.html#directly-accessing-the-summary-writer ''' def _on_training_start(self): self._log_freq = 10 # log every 10 calls output_formats = self.logger.output_formats # Save reference to tensorboard formatter object # note: the failure case (not formatter found) is not handled here, should be done with try/except. self.tb_formatter = next(formatter for formatter in output_formats if isinstance(formatter, TensorBoardOutputFormat)) def _on_step(self) -> bool: ''' Log my_custom_reward every _log_freq(th) to tensorboard for each environment ''' if self.n_calls % self._log_freq == 0: rewards = self.locals['my_custom_info_dict']['my_custom_reward'] for i in range(self.locals['env'].num_envs): self.tb_formatter.writer.add_scalar("rewards/env #{}".format(i+1), rewards[i], self.n_calls) if __name__ == "__main__": env_id = "CartPole-v1" envs = SubprocVecEnv([make_env(env_id, i) for i in range(4)]) # 4 environments model = SAC("MlpPolicy", envs, tensorboard_log="/tmp/sac/", verbose=1) model.learn(50000, callback=TensorboardCallback())
Question: Is there a short version of n = [5, 3, 17] if 5 in n and 17 in n: print("YES")  something like that doesn't seem to work if (5 and 17) in n: print("YES")  Any suggestions?

Answer: You could use something like this instead: n = [5,3,7] if all(item in n for item in [5,7]): print("YES")
Question: version pip 21.2.4 python 3.6 The command: pip install -r requirements.txt  The content of my requirements.txt: mongoengine==0.19.1 numpy==1.16.2 pylint pandas==1.1.5 fawkes  The command is failing with this error ERROR: Command errored out with exit status 1: command: /Users/*/Desktop/ml/*/venv/bin/python -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/kn/0y92g7x55qs7c42tln4gwhtm0000gp/T/pip-install-soh30mel/mongoengine_89e68f8427244f1bb3215b22f77a619c/setup.py'"'"'; __file__='"'"'/private/var/folders/kn/0y92g7x55qs7c42tln4gwhtm0000gp/T/pip-install-soh30mel/mongoengine_89e68f8427244f1bb3215b22f77a619c/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /private/var/folders/kn/0y92g7x55qs7c42tln4gwhtm0000gp/T/pip-pip-egg-info-97994d6e cwd: /private/var/folders/kn/0y92g7x55qs7c42tln4gwhtm0000gp/T/pip-install-soh30mel/mongoengine_89e68f8427244f1bb3215b22f77a619c/ Complete output (1 lines): error in mongoengine setup command: use_2to3 is invalid. ---------------------------------------- WARNING: Discarding https://*/pypi/packages/mongoengine-0.19.1.tar.gz#md5=68e613009f6466239158821a102ac084 (from https://*/pypi/simple/mongoengine/). Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output. ERROR: Could not find a version that satisfies the requirement mongoengine==0.19.1 (from versions: 0.15.0, 0.19.1) ERROR: No matching distribution found for mongoengine==0.19.1 

Answer: It looks like setuptools>=58 breaks support for use_2to3: setuptools changelog for v58 So you should update setuptools to setuptools<58 or avoid using packages with use_2to3 in the setup parameters. I was having the same problem, pip==19.3.1
Question: I was going through the python grammer specification and find the following statement  for_stmt: | 'for' star_targets 'in' ~ star_expressions ':' [TYPE_COMMENT] block [else_block]  What does ~ means in this grammar rule?. The other symbols used in the grammer(like &, !, |) are already documented but not ~.  The notation is a mixture of EBNF and PEG. In particular, & followed by a symbol, token or parenthesized group indicates a positive lookahead (i.e., is required to match but not consumed), while ! indicates a negative lookahead (i.e., is required not to match). We use the | separator to mean PEG’s “ordered choice” (written as / in traditional PEG grammars) 

Answer: It's documented in PEP 617 under Grammar Expressions:  ~ Commit to the current alternative, even if it fails to parse. rule_name: '(' ~ some_rule ')' | some_alt In this example, if a left parenthesis is parsed, then the other alternative won’t be considered, even if some_rule or ‘)’ fail to be parsed.  The ~ basically indicates that once you reach it, you're locked into the particular rule and cannot move onto the next rule if the parse fails. PEP 617 mentions earlier that | some_alt can be written in the next line.
Question: I am using jupyter notebook to create python notes(sort of) for a virtual lecture. I like to use vscode instead of jupyter lab. But unfortunately the font size of the markdown output is too small(to see on participants' screen on virtual call). While using jupyter lab, i used to zoom the whole browser. But i can't do that in vscode, also i can't find any setting that changes the font size for it. The setting to change font size only applies to the vs code editor window. Does anyone has any idea on how i can increase the size.  P.S.- I know that we could use # on each line to increase its size but i can't go around putting # before every line as i have to make a rather large document. And you get the idea how small the standard size is from the image.

Answer: A new setting is being added to vscode: notebook.markup.fontSize Should be in the Insiders Build v1.63 soon. See https://github.com/microsoft/vscode/issues/126294#issuecomment-964601412