eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I am trying to determine roc_auc_score for a fit model on a validation set. I am seeing some conflicting information on function inputs. Documentation says: "y_score array-like of shape (n_samples,) or (n_samples, n_classes) Target scores. In the binary and multilabel cases, these can be either probability estimates or non-thresholded decision values (as returned by decision_function on some classifiers). In the multiclass case, these must be probability estimates which sum to 1. The binary case expects a shape (n_samples,), and the scores must be the scores of the class with the greater label. The multiclass and multilabel cases expect a shape (n_samples, n_classes). In the multiclass case, the order of the class scores must correspond to the order of labels, if provided, or else to the numerical or lexicographical order of the labels in y_true." Not sure exactly what this calls for: 1) predicted probabilities against the actual y values in the test set or 2) class predictions against the actual y values in the test set I've been searching and, in the binary classification case (my interest), some people use predicted probabilities while others use actual predictions (0 or 1). In other words: Fit model: model.fit(X_train, y_train)  Use either: y_preds = model.predict(X_test)  or: y_probas = model.predict_proba(X_test)  I find that: roc_auc_score(y_test, y_preds)  and: roc_auc_score(y_test, y_probas[:,1]) # probabilites for the 1 class  yield very different results. Which one is correct? I also find that to actually plot the ROC Curve I need to use probabilities. Any guidance appreciated.

Answer: model.predict(...) will give you the predicted label for each observation. That is, it will return an array full of ones and zeros. model.predict_proba(...)[:, 1] will give you the probability for each observation being equal to one. That is, it will return an array full of numbers between zero and one, inclusive. A ROC curve is calculated by taking each possible probability, using it as a threshold and calculating the resulting True Positive and False Positive rates. Hence, if you pass model.predict(...) to metrics.roc_auc_score(), you are calculating the AUC for a ROC curve that only used two thresholds (either one or zero). This is incorrect, as these are not the predicted probabilities of your model. To get the AUC of your model, you need to pass the predicted probabilities to roc_auc_score(...): from sklearn.metrics import roc_auc_score roc_auc_score(y_test, model.predict_proba(X_test)[:, 1])
Question: I am new to encryption and hashing in python. I need this for authentication in one of my flask project. So my friend told me to use crypto package but when i searched it up i got crypto and pycrypto packages in the result. The thing is I know both of them are for encryption utility but I am confused as to which one to use. Is one of them better than the other or is one of them just a wrapper of the other? Or is there another better encryption package for python that I should use instead of the two mentioned above? I hope someone who have used these packages could help me. Thanks.

Answer: These two packages serve very different goals: crypto is a command line utility, which is intended to encrypt files, while pycrypto is a Python library which can be used from within Python to perform a number of different cryptographic operations (hashing, encryption/decryption, etc). pycrypto would be the more appropriate choice for implementing authentication within Python. I will note that Python also includes some cryptographic primitives in the standard library, which may be more suitable for your use case. Edit: As has been noted in the comments, pycrypto is no longer maintained, and a library such as cryptography or pycryptodome should be used instead.
Question: I have this code: import requests r = requests.get('https://www.instagram.com/p/CJDxE7Yp5Oj/?__a=1') data = r.json()['graphql']['shortcode_media']  Why do I get an error like this? C:\ProgramData\Anaconda3\envs\test\python.exe C:/Users/Solba/PycharmProjects/test/main.py Traceback (most recent call last): File "C:/Users/Solba/PycharmProjects/test/main.py", line 4, in <module> data = r.json() File "C:\ProgramData\Anaconda3\envs\test\lib\site-packages\requests\models.py", line 900, in json return complexjson.loads(self.text, **kwargs) File "C:\ProgramData\Anaconda3\envs\test\lib\json\__init__.py", line 357, in loads return _default_decoder.decode(s) File "C:\ProgramData\Anaconda3\envs\test\lib\json\decoder.py", line 337, in decode obj, end = self.raw_decode(s, idx=_w(s, 0).end()) File "C:\ProgramData\Anaconda3\envs\test\lib\json\decoder.py", line 355, in raw_decode raise JSONDecodeError("Expecting value", s, err.value) from None json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0) Process finished with exit code 1 

Answer: r.json() expects a JSON string to be returned by the API. The API should explicitly say it is responding with JSON through response headers. In this case, the URL you are requesting is either not responding with a proper JSON or not explicitly saying it is responding with a JSON. You can first check the response sent by the URL by: data = r.text print(data)  If the response can be treated as a JSON string, then you can process it with: import json data = json.loads(r.text)  Note: You can also check the content-type and Accept headers to ensure the request and response are in the required datatype
Question: I can get mydrive folders, but I can't access shared drive folders from Google Drive API. This is my code.(almost same to the Guides' code here) I followed the Guides, finished "Enable the Drive API", execute the pip command on VScode, and put credentials.json to the working directory. (I got no error, only got filename list of mydrive or 'No files found' that is printed by the code.) from __future__ import print_function import pickle import os.path from googleapiclient.discovery import build from google_auth_oauthlib.flow import InstalledAppFlow from google.auth.transport.requests import Request # If modifying these scopes, delete the file token.pickle. SCOPES = ['https://www.googleapis.com/auth/drive.metadata.readonly'] def main(): """Shows basic usage of the Drive v3 API. Prints the names and ids of the first 10 files the user has access to. """ creds = None # The file token.pickle stores the user's access and refresh tokens, and is # created automatically when the authorization flow completes for the first # time. if os.path.exists('token.pickle'): with open('token.pickle', 'rb') as token: creds = pickle.load(token) # If there are no (valid) credentials available, let the user log in. if not creds or not creds.valid: if creds and creds.expired and creds.refresh_token: creds.refresh(Request()) else: flow = InstalledAppFlow.from_client_secrets_file( 'credentials.json', SCOPES) creds = flow.run_local_server(port=0) # Save the credentials for the next run with open('token.pickle', 'wb') as token: pickle.dump(creds, token) service = build('drive', 'v3', credentials=creds) # Call the Drive v3 API results = service.files().list( pageSize=10, fields="nextPageToken, files(id, name)").execute() fields="nextPageToken, files(id, name)").execute() items = results.get('files', []) if not items: print('No files found.') else: print('Files:') for item in items: print(u'{0} ({1})'.format(item['name'], item['id'])) if __name__ == '__main__': main() 

Answer: Notice that the API has the includeItemsFromAllDrives parameter in order to determine whether shared drive items show up or not in the results. The Python API V3 wrapper also has this parameter included on it's list method implementation that needs to be included when calling the list() method: ... service = build('drive', 'v3', credentials=creds) # Call the Drive v3 API results = service.files().list( pageSize=10, includeItemsFromAllDrives=True, supportsAllDrives=True, fields="nextPageToken, files(id, name)").execute() items = results.get('files', []) ...
Question: I'm trying to insert a png image in matplotlib figure (ref) import matplotlib.pyplot as plt import numpy as np from matplotlib.figure import Figure from matplotlib.offsetbox import OffsetImage, AnnotationBbox ax = plt.subplot(111) ax.plot( [1, 2, 3], [1, 2, 3], 'go-', label='line 1', linewidth=2 ) arr_img = plt.imread("stinkbug.png") im = OffsetImage(arr_img) ab = AnnotationBbox(im, (1, 0), xycoords='axes fraction') ax.add_artist(ab) plt.show()  Inset image:  Output obtained:  I'd like to know how to resize the image that has to be inserted to avoid overlaps. EDIT: Saving the figure ax.figure.savefig("output.svg", transparent=True, dpi=600, bbox_inches="tight")  

Answer: You can zoom the image and the set the box alignment to the lower right corner (0,1) plus some extra for the margins: im = OffsetImage(arr_img, zoom=.45) ab = AnnotationBbox(im, (1, 0), xycoords='axes fraction', box_alignment=(1.1,-0.1))   You may also want to use data coordinates, which is the default, and use the default box_alignment to the center, e.g. ab = AnnotationBbox(im, (2.6, 1.45)). See the xycoords parameter doc for more information about various coordinate options.
Question: I have a script that's two directories down. ❯ tree . └── foo └── bar └── test.py ❯ cd foo/bar ❯ cat test.py from pathlib import Path print(Path(__file__).parent) print(Path(__file__).parent.parent)  When I run it from the directory that contains it, PathLib thinks that the file's grandparent is the same as its parent. ❯ python test.py . # <-- same . # <-- directories  But when I run it from the top level, PathLib behaves correctly. ❯ cd ../.. ❯ python foo/bar/test.py foo/bar # <-- different foo # <-- directories  Am I misunderstanding something about PathLib's API, or is something else causing its output to be sensitive to my working directory?

Answer: You need to call Path.resolve() to make your path absolute (a full path including all parent directories and removing all symlinks) from pathlib import Path print(Path(__file__).resolve().parent) print(Path(__file__).resolve().parent.parent)  This will cause the results to include the entire path to each directory, but the behaviour will work wherever it is called from
Question: I have users that create both conda and pip packages- I have no control over this I use artifactory to host private conda and pip repos, for example this is how a private pip repo works: https://www.jfrog.com/confluence/display/JFROG/PyPI+Repositories Sometimes there is a private pip package a conda environment or package needs. How can I configure conda to get my private pip packages from my private repo? I haven't found documentation on this. I would like for this to be transparent for users as much as possible- so they set up their config once and in their conda environment they can easily specify a private pip package and it just works

Answer: Conda won't search PyPI or alternative pip-compatible indexes automatically, but one can still use the --index-url or --extra-index-url flags when using pip install. E.g., Ad Hoc Installation # activate environment conda activate foo # ensure it has `pip` installed conda list pip # install with `pip` pip install --extra-index-url http://localhost:8888 bar  YAML-based environments foo.yaml name: foo channels: - defaults dependencies: - python - pip - pip: - --extra-index-url http://localhost:8888 - bar  Environment creation conda env create -f foo.yaml
Question: I can return the frequency of all columns in a nice dataframe with a total column. for column in df: df.groupby(column).size().reset_index(name="total") Count total 0 1 423 1 2 488 2 3 454 3 4 408 4 5 343 Precipitation total 0 Fine 7490 1 Fog 23 2 Other 51 3 Raining 808 Month total 0 1 717 1 2 648 2 3 710 3 4 701  I put the loop in a function, but this returns the first column "Count" only. def count_all_columns_freq(dataframe_x): for column in dataframe_x: return dataframe_x.groupby(column).size().reset_index(name="total") count_all_columns_freq(df) Count total 0 1 423 1 2 488 2 3 454 3 4 408 4 5 343  Is there a way to do this using slicing or other method e.g. for column in dataframe_x[1:]:

Answer: Based on your comment, you just want to return a list of dataframe: def count_all_columns_freq(df): return [df.groupby(column).size().reset_index(name="total") for column in df]  You can select columns in many ways in pandas, e.g. by slicing or by passing a list of columns like in df[['colA', 'colB']]. You don't need to change the function for that. Personally, I would return a dictionary instead: def frequency_dict(df): return {column: df.groupby(column).size() for column in df} # so that I could use it like this: freq = frequency_dict(df) freq['someColumn'].loc[value]  EDIT: "What if I want to count the number of NaN?" In that case, you can pass dropna=False to groupby (this works for pandas >= 1.1.0): def count_all_columns_freq(df): return [df.groupby(column, dropna=False).size().reset_index(name="total") for column in df]
Question: So, I was just messing around with pyautogui moving the mouse to random positions on the screen, when I manually moved the mouse to the top left corner of the screen and ran the program, it raised a pyautogui failsafe. I do know how to disable it and all that, but I want to know why is is there in the first place and possible use cases Code: import pyautogui pyautogui.click(x=25, y=1048) time.sleep(2) # I moved the move to the corner of the screen during this time delay pyautogui.click(x=701, y=430)  Error: Traceback (most recent call last): File "C:\1 Files and Folders\folder\Python Project\Python\CODE\My Projects\Automation.py", line 23, in <module> job() File "C:\1 Files and Folders\folder\Python Project\Python\CODE\My Projects\Automation.py", line 18, in job pyautogui.click(x=1475, y=141) File "C:\Users\My_user\AppData\Local\Programs\Python\Python39-32\lib\site-packages\pyautogui\__init__.py", line 585, in wrapper failSafeCheck() File "C:\Users\My_user\AppData\Local\Programs\Python\Python39-32\lib\site-packages\pyautogui\__init__.py", line 1710, in failSafeCheck raise FailSafeException( pyautogui.FailSafeException: PyAutoGUI fail-safe triggered from mouse moving to a corner of the screen. To disable this fail-safe, set pyautogui.FAILSAFE to False. DISABLING FAIL-SAFE IS NOT RECOMMENDED. 

Answer: According to pyautogui docs  It’s hard to use the mouse to close a program if the mouse cursor is moving around on its own. As a safety feature, a fail-safe feature is enabled by default. When a PyAutoGUI function is called, if the mouse is in any of the four corners of the primary monitor, they will raise a pyautogui.FailSafeException.  So simply FailSafe is just thing that if your program just spams the mouse or doing something that you cannot stop it, You can close that in this way, For more information read it docs pyautogui docs
Question: I have a dataframe. I have aggregated as below. But, I want to difference them as max value - min values  dnm=df.groupby('Type').agg({'Vehicle_Age': ['max','min']})  Expect: 

Answer: You can use np.ptp, this does the max - min calculation for you: df.groupby('Type').agg({'Vehicle_Age': np.ptp})  Or, df.groupby('Type')['Vehicle_Age'].agg(np.ptp)  If you a Series as the output.
Question: I have what should've been a simple task, and it has stumped me for a while. I am trying to patch an object imported into the current module. Per the answers to Mock patching from/import statement in Python I should just be able to patch("__main__.imported_obj"). However, this isn't working for me. Please see my below minimal repro (I am running the tests via pytest): Minimal Repro This is run using Python 3.8.6. from random import random from unittest.mock import patch import pytest @pytest.fixture def foo(): with patch("__main__.random"): return def test(foo) -> None: pass  When I run this code using PyCharm, I get an AttributeError: AttributeError: <module '__main__' from '/Applications/PyCharm.app/Contents/plugins/python/helpers/pycharm/_jb_pytest_runner.py'> does not have the attribute 'random'  Furthermore, when I enter debugger mode in the line before the with patch, I see the attribute __main__ is not defined. I am not sure if it needs to be defined for patch to work its magic. NOTE: I know I can use patch.object and it becomes much easier. However, I am trying to figure out how to use patch in this question. Research Unable to mock open, even when using the example from the documentation This question is related because it's both a similar error message and use case. Their solution was to use builtins instead of __main__, but that's because they were trying to patch a built-in function (open).

Answer: You are assuming that the module the test is running in is __main__, but that would only be the case if it were called via main. This is usually the case if you are using unittest. With pytest, the tests live in the module they are defined in. You have to patch the current module, the name of which is accessible via __name__, instead of assuming a specific module name: from random import random from unittest.mock import patch import pytest @pytest.fixture def foo(): with patch(__name__ + ".random"): yield
Question: So I'm currently working on a python scraper to collect website info with selenium in python. The issue I'm having is if I head to a page that isn't live I get the error: unknown error: net::ERR_NAME_NOT_RESOLVED I haven't used python in a while so my knowledge isn't the best. Here is my code driver = webdriver.Chrome(ChromeDriverManager().install()) try: driver.get('%s' %link) except ERR_NAME_NOT_RESOLVED: print ("page down")  example website: http://www.whitefoxcatering.co.uk error Traceback (most recent call last): File "C:\Users\STE\AppData\Local\Programs\Python\Python39\selenium email\selenium test.py", line 106, in <module> driver.get('%s' %url) File "C:\Users\STE\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.8_qbz5n2kfra8p0\LocalCache\local-packages\Python38\site-packages\selenium\webdriver\remote\webdriver.py", line 333, in get self.execute(Command.GET, {'url': url}) File "C:\Users\STE\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.8_qbz5n2kfra8p0\LocalCache\local-packages\Python38\site-packages\selenium\webdriver\remote\webdriver.py", line 321, in execute self.error_handler.check_response(response) File "C:\Users\STE\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.8_qbz5n2kfra8p0\LocalCache\local-packages\Python38\site-packages\selenium\webdriver\remote\errorhandler.py", line 242, in check_response raise exception_class(message, screen, stacktrace) selenium.common.exceptions.WebDriverException: Message: unknown error: net::ERR_NAME_NOT_RESOLVED (Session info: chrome=87.0.4280.88)  I've searched through the previous questions and cannot seem to find a solution to this. Any help would be hugely appreciated :)

Answer: If you want to catch the error you have to change your except to what selenium is raising for you. In this case: selenium.common.exceptions.WebDriverException. So first import it: from selenium.common.exceptions import WebDriverException  Then you can catch: try: driver.get('http://www.whitefoxcatering.co.uk') except WebDriverException: print("page down")
Question: I am trying to update the version of infixpy using twine. Here is my ~/.pypirc: index-servers = pypi pypitest [pypi] repository: https://upload.pypi.org/legacy/ username: myuser password: mypassword [pypitest] repository: https://upload.testpypi.org/legacy username: myuser password: mypassword  Here is the command line: python setup.py build twine upload -r pypi dist/  The upload errors out with InvalidDistribution: Unknown distribution format: '' Here is the full output: Processing dependencies for infixpy==0.0.6 Finished processing dependencies for infixpy==0.0.6 Processing /Users/steve/git/infixpy Building wheels for collected packages: infixpy Building wheel for infixpy (setup.py) ... done Created wheel for infixpy: filename=infixpy-0.0.6-py3-none-any.whl size=43459 sha256=01fed46f42fa86475079636a55685c93521989aa0ba6558726a9d35c01004b7a Stored in directory: /private/var/folders/d6/m67jyndd7h754m3810cl3bpm0000gp/T/pip-ephem-wheel-cache-1bizg6_y/wheels/47/66/74/d79a56979feba04c8ef05e12fe861cacf813cecd397e57071f Successfully built infixpy Installing collected packages: infixpy Attempting uninstall: infixpy Found existing installation: infixpy 0.0.6 Uninstalling infixpy-0.0.6: Successfully uninstalled infixpy-0.0.6 Successfully installed infixpy-0.0.6 Uploading distributions to https://upload.pypi.org/legacy/ InvalidDistribution: Unknown distribution format: ''  What corrections are needed to my publishing process? I am on Python 3.7 on macOS.

Answer: Per the docs for twine upload (emphasis mine):  positional arguments: dist The distribution files to upload to the repository (package index). Usually dist/* . May additionally contain a .asc file to include an existing signature with the file upload.  You've passed a directory, not files - as the docs suggest, you probably want dist/*. If you pass a directory there are no matches for known distributions, as these are based on file extension, so you end up at this error case:  else: raise exceptions.InvalidDistribution( "Unknown distribution format: '%s'" % os.path.basename(filename) )  The basename for a directory is '', hence the not-so-helpful output.
Question: I want to exclude all the Optional values that are not set when I create JSON. In this example: from pydantic import BaseModel from typing import Optional class Foo(BaseModel): x: int y: int = 42 z: Optional[int] print(Foo(x=3).json())  I get {"x": 3, "y": 42, "z": null}. But I would like to exclude z. Not because its value is None, but because it is Optional and there was no keyword argument for z. In the two cases below I would like to have z in the JSON. Foo(x=1, z=None) Foo(x=1, z=77)  If there is any other solution to set z to optional in this sense, I would like to see it.

Answer: You could exclude only optional model fields that unset by making of union of model fields that are set and those that are not None. Pydantic provides the following arguments for exporting method model.dict(...):  exclude_unset: whether fields which were not explicitly set when creating the model should be excluded from the returned dictionary; default False. exclude_none: whether fields which are equal to None should be excluded from the returned dictionary; default False  To make union of two dicts we can use the expression a = {**b, **c} (values from c overwrites values from b). Note that since Python 3.9 it could be done just as a = b | c. from pydantic import BaseModel from typing import Optional from pydantic.json import pydantic_encoder import json class Foo(BaseModel): x: int y: int = 42 z: Optional[int] def exclude_optional_dict(model: BaseModel): return {**model.dict(exclude_unset=True), **model.dict(exclude_none=True)} def exclude_optional_json(model: BaseModel): return json.dumps(exclude_optional_dict(model), default=pydantic_encoder) print(exclude_optional_json(Foo(x=3))) # {"x": 3, "y": 42} print(exclude_optional_json(Foo(x=3, z=None))) # {"x": 3, "z": null, "y": 42} print(exclude_optional_json(Foo(x=3, z=77))) # {"x": 3, "z": 77, "y": 42}  Update In order for the approach to work with nested models, we need to do a deep union( or merge) of two dictionaries, like so: def union(source, destination): for key, value in source.items(): if isinstance(value, dict): node = destination.setdefault(key, {}) union(value, node) else: destination[key] = value return destination def exclude_optional_dict(model: BaseModel): return union(model.dict(exclude_unset=True), model.dict(exclude_none=True)) class Foo(BaseModel): x: int y: int = 42 z: Optional[int] class Bar(BaseModel): a: int b: int = 52 c: Optional[int] d: Foo print(exclude_optional_json(Bar(a=4, d=Foo(x=3)))) print(exclude_optional_json(Bar(a=4, c=None, d=Foo(x=3, z=None)))) print(exclude_optional_json(Bar(a=4, c=78, d=Foo(x=3, z=77))))  {"a": 4, "b": 52, "d": {"x": 3, "y": 42}} {"a": 4, "b": 52, "d": {"x": 3, "y": 42, "z": null}, "c": null} {"a": 4, "b": 52, "c": 78, "d": {"x": 3, "y": 42, "z": 77}}