eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I would like to register a dataset from ADLS Gen2 in my Azure Machine Learning workspace (azureml-core==1.12.0). Given that service principal information is not required in the Python SDK documentation for .register_azure_data_lake_gen2(), I successfully used the following code to register ADLS gen2 as a datastore: from azureml.core import Datastore adlsgen2_datastore_name = os.environ['adlsgen2_datastore_name'] account_name=os.environ['account_name'] # ADLS Gen2 account name file_system=os.environ['filesystem'] adlsgen2_datastore = Datastore.register_azure_data_lake_gen2( workspace=ws, datastore_name=adlsgen2_datastore_name, account_name=account_name, filesystem=file_system )  However, when I try to register a dataset, using from azureml.core import Dataset adls_ds = Datastore.get(ws, datastore_name=adlsgen2_datastore_name) data = Dataset.Tabular.from_delimited_files((adls_ds, 'folder/data.csv'))  I get an error  Cannot load any data from the specified path. Make sure the path is accessible and contains data. ScriptExecutionException was caused by StreamAccessException. StreamAccessException was caused by AuthenticationException. 'AdlsGen2-ReadHeaders' for '[REDACTED]' on storage failed with status code 'Forbidden' (This request is not authorized to perform this operation using this permission.), client request ID <CLIENT_REQUEST_ID>, request ID <REQUEST_ID>. Error message: [REDACTED] | session_id=<SESSION_ID>  Do I need the to enable the service principal to get this to work? Using the ML Studio UI, it appears that the service principal is required even to register the datastore. Another issue I noticed is that AMLS is trying to access the dataset here: https://adls_gen2_account_name.**dfs**.core.windows.net/container/folder/data.csv whereas the actual URI in ADLS Gen2 is: https://adls_gen2_account_name.**blob**.core.windows.net/container/folder/data.csv

Answer: According to this documentation,you need to enable the service principal. 1.you need to register your application and grant the service principal with Storage Blob Data Reader access.  2.try this code: adlsgen2_datastore = Datastore.register_azure_data_lake_gen2(workspace=ws, datastore_name=adlsgen2_datastore_name, account_name=account_name, filesystem=file_system, tenant_id=tenant_id, client_id=client_id, client_secret=client_secret ) adls_ds = Datastore.get(ws, datastore_name=adlsgen2_datastore_name) dataset = Dataset.Tabular.from_delimited_files((adls_ds,'sample.csv')) print(dataset.to_pandas_dataframe())  Result:
Question: Since the new update of azure-storage-blob, the blockblobservice is depreciated How can I check that a blob exist ? This answer is not working with the new version of azure-storage-blob Faster Azure blob name search with python? I found this issue on GitHub : https://github.com/Azure/azure-sdk-for-python/issues/12744

Answer: Version 12.5.0 released on 2020-09-10 has now the exists method in the new SDK. For example, Sync: from azure.storage.blob import BlobClient blob = BlobClient.from_connection_string(conn_str="my_connection_string", container_name="mycontainer", blob_name="myblob") exists = blob.exists() print(exists)  Async: import asyncio async def check(): from azure.storage.blob.aio import BlobClient blob = BlobClient.from_connection_string(conn_str="my_connection_string", container_name="mycontainer", blob_name="myblob") async with blob: exists = await blob.exists() print(exists)
Question: According to tutorialspoint.com, Python is a functional programming language. "Some of the popular functional programming languages include: Lisp, Python, Erlang, Haskell, Clojure, etc." https://www.tutorialspoint.com/functional_programming/functional_programming_introduction.htm But other sources say Python is an object-oriented programming language (you can create objects in Python). So is Python both? If so, if you're trying to program something that requires lots of mathematical computations, would Python still be a good choice (Since functional languages have concurrency, better syntax for math, and higher-level functions)?

Answer: Python, like many others, is a multi-paradigm language. You can use it as a fairly strictly imperative language, you can use it in a more object-oriented way, and you can use it in a more functional way. One important thing to note though is that functional is generally contrasted with imperative, object-oriented tends to exist at a different level and can be "layered over" a more imperative or a more functional core. However Python is largely an imperative and object oriented language: much of the builtins and standard library are really built around classes and objects, and it doesn't encourage the sort of thinking which functional languages generally drive the user to. In fact going through the (fairly terrible) list the article you link to provides, Python lands on the OOP side of more or less all of them:  it doesn't use immutable data much (it's not really possible to define immutable types in pure python, most of the collections are mutable, and the ones which are not are not designed for functional updates) its execution model is very imperative it has limited support for parallel programming its functions very much do have side-effects flow control is absolutely not done using function calls it's not a language which encourages recursion execution order is very relevant and quite strictly defined  Then again, much of the article is nonsense. If that is typical of that site, I'd recommend using something else.  If so, if you're trying to program something very mathematical and computational, would Python still be a good choice  Well Python is a pretty slow language in and of itself, but at the same time it has a very large and strong ecosystem of scientific libraries. It's probably not the premier language for abstract mathematics (it's rather bad at symbolic manipulation) but it tends to be a relatively good glue or prototyping tool.  As functional languages are more suitable for mathematical stuff  Not necessarily. But not knowing what you actually mean by "mathematical stuff" it's hard to judge. Do you mean symbolic manipulations? Statistics? Hard computations? Something else entirely?
Question: I try to test an endpoint with the TestClient from FastAPI (which is the Scarlett TestClient basically). The response code is always 422 Unprocessable Entity. This is my current Code: from typing import Dict, Optional from fastapi import APIRouter from pydantic import BaseModel router = APIRouter() class CreateRequest(BaseModel): number: int ttl: Optional[float] = None @router.post("/create") async def create_users(body: CreateRequest) -> Dict: return { "msg": f"{body.number} Users are created" }  As you can see I'm also passing the application/json header to the client to avoid a potential error. And this is my Test: from fastapi.testclient import TestClient from metusa import app def test_create_50_users(): client = TestClient(app) client.headers["Content-Type"] = "application/json" body = { "number": 50, "ttl": 2.0 } response = client.post('/v1/users/create', data=body) assert response.status_code == 200 assert response.json() == {"msg": "50 Users created"}  I also found this error message in the Response Object b'{"detail":[{"loc":["body",0],"msg":"Expecting value: line 1 column 1 (char 0)","type":"value_error.jsondecode","ctx":{"msg":"Expecting value","doc":"number=50&ttl=2.0","pos":0,"lineno":1,"colno":1}}]}'  Thank you for your support and time!

Answer: You don't need to set headers manualy. You can use json argument insteed of data in client.post method. def test_create_50_users(): client = TestClient(router) body = { "number": 50, "ttl": 2.0 } response = client.post('/create', json=body)  If you still want to use data attribute, you need to use json.dumps def test_create_50_users(): client = TestClient(router) client.headers["Content-Type"] = "application/json" body = { "number": 50, "ttl": 2.0 } response = client.post('/create', data=json.dumps(body))
Question: I am trying to run below code on Jupyter Notebook: lr = LogisticRegression(class_weight='balanced') lr.fit(X_train,y_train) y_pred = lr.predict(X_train) acc_log = round(lr.score(X_train, y_train) * 100, 2) prec_log = round(precision_score(y_train,y_pred) * 100,2) recall_log = round(recall_score(y_train,y_pred) * 100,2) f1_log = round(f1_score(y_train,y_pred) * 100,2) roc_auc_log = roc_auc_score(y_train,y_pred)  When trying to execute this, I am getting the below error: --------------------------------------------------------------------------- TypeError Traceback (most recent call last) <ipython-input-46-bcb2d9729eb6> in <module> 6 prec_log = round(precision_score(y_train,y_pred) * 100,2) 7 recall_log = round(recall_score(y_train,y_pred) * 100,2) ----> 8 f1_log = round(f1_score(y_train,y_pred) * 100,2) 9 roc_auc_log = roc_auc_score(y_train,y_pred) TypeError: 'numpy.float64' object is not callable  Can't seem to figure out what I am doing wrong.

Answer: Somewhere in your code (not shown here), there is a line which says f1_score = ... (with the written type being numpy.float64) so you're overriding the method f1_score with a variable f1_score (which is not callable, hence the error message). Rename one of the two to resolve the error.
Question: I'm trying to serialize a torch tensor using protobuf and it seems using BytesIO along with torch.save() doesn't work. I have tried: import torch import io x = torch.randn(size=(1,20)) buff = io.BytesIO() torch.save(x, buff) print(f'buffer: {buff.read()}')  to no avail as it results in b'' in the output! How should I be going about this?

Answer: You need to seek to the beginning of the buffer before reading: import torch import io x = torch.randn(size=(1,20)) buff = io.BytesIO() torch.save(x, buff) buff.seek(0) # <-- this is what you were missing print(f'buffer: {buff.read()}')  gives you this magnificent output:  buffer: b'PK\x03\x04\x00\x00\x08\x08\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x10\x00\x12\x00archive/data.pklFB\x0e\x00ZZZZZZZZZZZZZZ\x80\x02ctorch._utils\n_rebuild_tensor_v2\nq\x00((X\x07\x00\x00\x00storageq\x01ctorch\nFloatStorage\nq\x02X\x0f\x00\x00\x00140417054790352q\x03X\x03\x00\x00\x00cpuq\x04K\x14tq\x05QK\x00K\x01K\x14\x86q\x06K\x14K\x01\x86q\x07\x89ccollections\nOrderedDict\nq\x08)Rq\ttq\nRq\x0b.PK\x07\x08\xf3\x08u\x13\xa8\x00\x00\x00\xa8\x00\x00\x00PK\x03\x04\x00\x00\x08\x08\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x1c\x00\x0e\x00archive/data/140417054790352FB\n\x00ZZZZZZZZZZ\xba\xf3x?\xb5\xe2\xc4=)R\x89\xbfM\x08\x19\xbfo%Y\xbf\x05\xc0_\xbf\x03N4\xbe\xdd_ \xc0&\xc4\xb5?\xa7\xfd\xc4?f\xf1$?Ll\xa6?\xee\x8e\x80\xbf\x88Uq?.<\xd8?{\x08\xb2?\xb3\xa3\xba>q\xcd\xbc?\xba\xe3h\xbd\xcan\x11\xc0PK\x07\x08A\xf3\xdc>P\x00\x00\x00P\x00\x00\x00PK\x03\x04\x00\x00\x08\x08\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x0f\x003\x00archive/versionFB/\x00ZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZ3\nPK\x07\x08\xd1\x9egU\x02\x00\x00\x00\x02\x00\x00\x00PK\x01\x02\x00\x00\x00\x00\x08\x08\x00\x00\x00\x00\x00\x00\xf3\x08u\x13\xa8\x00\x00\x00\xa8\x00\x00\x00\x10\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00archive/data.pklPK\x01\x02\x00\x00\x00\x00\x08\x08\x00\x00\x00\x00\x00\x00A\xf3\xdc>P\x00\x00\x00P\x00\x00\x00\x1c\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\xf8\x00\x00\x00archive/data/140417054790352PK\x01\x02\x00\x00\x00\x00\x08\x08\x00\x00\x00\x00\x00\x00\xd1\x9egU\x02\x00\x00\x00\x02\x00\x00\x00\x0f\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\xa0\x01\x00\x00archive/versionPK\x06\x06,\x00\x00\x00\x00\x00\x00\x00\x1e\x03-\x00\x00\x00\x00\x00\x00\x00\x00\x00\x03\x00\x00\x00\x00\x00\x00\x00\x03\x00\x00\x00\x00\x00\x00\x00\xc5\x00\x00\x00\x00\x00\x00\x00\x12\x02\x00\x00\x00\x00\x00\x00PK\x06\x07\x00\x00\x00\x00\xd7\x02\x00\x00\x00\x00\x00\x00\x01\x00\x00\x00PK\x05\x06\x00\x00\x00\x00\x03\x00\x03\x00\xc5\x00\x00\x00\x12\x02\x00\x00\x00\x00'
Question: I tried drawing a relplot with log scaled axes. Making use of previous answers, I tried: import matplotlib.pyplot as plt import seaborn as sns f, ax = plt.subplots(figsize=(7, 7)) ax.set(xscale="log", yscale="log") tips = sns.load_dataset("tips") sns.relplot(x="total_bill", y="tip", hue='smoker', data=tips) plt.show()  However the axes were not changed in the result.  How can I remedy this?

Answer: You can use scatterplot and dont forget to mention your axes in your plot import matplotlib.pyplot as plt import seaborn as sns f, ax = plt.subplots(figsize=(7, 7)) tips = sns.load_dataset("tips") ax.set(xscale="log", yscale="log") sns.scatterplot(x="total_bill", y="tip", hue='smoker', data=tips,ax=ax) plt.show()  Edit - relplot is a figure-level function and does not accept the ax= paramter
Question: I have the following dataframe in pandas: >>>name food beverage age 0 Ruth Burger Cola 23 1 Dina Pasta water 19 2 Joel Tuna water 28 3 Daniel null soda 30 4 Tomas null cola 10  I want to put condistion that if value in food column is null, the age and beverage will change into ' ' (blank as well), I have wrote this code for that: if df[(df['food'].isna())]: df['beverage']=' ' df['age']=' '  but I keep getting error:  ValueError: The truth value of a DataFrame is ambiguous. Use a.empty, a.bool(), a.item(), a.any() or a.all().  I have playes with the location of the ([ but didn't help, what do I do wrong?

Answer: Try with mask df[['beverage','age']] = df[['beverage','age']].mask(df['food'].isna(),'') df Out[86]: name food beverage age 0 Ruth Burger Cola 23 1 Dina Pasta water 19 2 Joel Tuna water 28 3 Daniel NaN 4 Tomas NaN
Question: So I need to have some routes inside a class, but the route methods need to have the self attr (to access the class' attributes). However, FastAPI then assumes self is its own required argument and puts it in as a query param This is what I've got: app = FastAPI() class Foo: def __init__(y: int): self.x = y @app.get("/somewhere") def bar(self): return self.x  However, this returns 422 unless you go to /somewhere?self=something. The issue with this, is that self is then str, and thus useless. I need some way that I can still access self without having it as a required argument.

Answer: For creating class-based views you can use @cbv decorator from fastapi-utils. The motivation of using it:  Stop repeating the same dependencies over and over in the signature of related endpoints.  Your sample could be rewritten like this: from fastapi import Depends, FastAPI from fastapi_utils.cbv import cbv from fastapi_utils.inferring_router import InferringRouter def get_x(): return 10 app = FastAPI() router = InferringRouter() # Step 1: Create a router @cbv(router) # Step 2: Create and decorate a class to hold the endpoints class Foo: # Step 3: Add dependencies as class attributes x: int = Depends(get_x) @router.get("/somewhere") def bar(self) -> int: # Step 4: Use `self.<dependency_name>` to access shared dependencies return self.x app.include_router(router)
Question: In my current project, I am interested in calculating spatially correlated noise for a large model grid. The noise should be strongly correlated over short distances, and uncorrelated over large distances. My current approach uses multivariate Gaussians with a covariance matrix specifying the correlation between all cells. Unfortunately, this approach is extremely slow for large grids. Do you have a recommendation of how one might generate spatially correlated noise more efficiently? (It doesn't have to be Gaussian) import scipy.stats import numpy as np import scipy.spatial.distance import matplotlib.pyplot as plt # Create a 50-by-50 grid; My actual grid will be a LOT larger X,Y = np.meshgrid(np.arange(50),np.arange(50)) # Create a vector of cells XY = np.column_stack((np.ndarray.flatten(X),np.ndarray.flatten(Y))) # Calculate a matrix of distances between the cells dist = scipy.spatial.distance.pdist(XY) dist = scipy.spatial.distance.squareform(dist) # Convert the distance matrix into a covariance matrix correlation_scale = 50 cov = np.exp(-dist**2/(2*correlation_scale)) # This will do as a covariance matrix # Sample some noise !slow! noise = scipy.stats.multivariate_normal.rvs( mean = np.zeros(50**2), cov = cov) # Plot the result plt.contourf(X,Y,noise.reshape((50,50))) 

Answer: Faster approach:  Generate spatially uncorrelated noise. Blur with Gaussian filter kernel to make noise spatially correlated.  Since the filter kernel is rather large, it is a good idea to use a convolution method based on Fast Fourier Transform. import numpy as np import scipy.signal import matplotlib.pyplot as plt # Compute filter kernel with radius correlation_scale (can probably be a bit smaller) correlation_scale = 50 x = np.arange(-correlation_scale, correlation_scale) y = np.arange(-correlation_scale, correlation_scale) X, Y = np.meshgrid(x, y) dist = np.sqrt(X*X + Y*Y) filter_kernel = np.exp(-dist**2/(2*correlation_scale)) # Generate n-by-n grid of spatially correlated noise n = 50 noise = np.random.randn(n, n) noise = scipy.signal.fftconvolve(noise, filter_kernel, mode='same') plt.contourf(np.arange(n), np.arange(n), noise) plt.savefig("fast.png")  Sample output of this method:  Sample output of slow method from question:  Image size vs running time:
Question: I see like this %(asctime)s in the logging module What is the meaning of %()s instead of %s? I only know %s means "string" and I can't find other information about %()s on the internet.

Answer: This is a string formatting feature when using the % form of Python string formatting to insert values into a string. The case you're looking at allows named values to be taken from a dictionary by providing the dictionary and specifying keys into that dictionary in the format string. Here's an example: values = {'city': 'San Francisco', 'state': 'California'} s = "I live in %(city)s, %(state)s" % values print(s)  Result: I live in San Francisco, California
Question: I want to enforce that all items in a list are of type x. What would be the best way to do this? Currently I am doing an assert like the following: a = [1,2,3,4,5] assert len(a) == len([i for i in a if isinstance(i, int)])  Where int is the type I'm trying to enforce here. Is there a better way to do this?

Answer: I think you are making it a little too complex. You can just use all(): a = [1,2,3,4,5] assert all(isinstance(i, int) for i in a) a = [1,2,3,4,5.5] assert all(isinstance(i, int) for i in a) # AssertionError
Question: I'd like to specify the versions of tensorflow in a Python module. The agreeable versions are: (version >= 1.14.0 and version < 2.0) or (version >= 2.2)  Does anyone know how to express this strange situation in a requirements.txt file? I believe there's a syntax for forbidding specific versions of a module, but I haven't been able to find it...

Answer: From PEP 440 Version Specifiers: tensorflow >=1.14.0,!=2.0.*,!= 2.1.*  The comma , represents a logical and. Note that requirements.txt files are used for pinning a deployment, I would generally only expect to ever see == specifiers used in those files.
Question: I have login history data from User A for a day. My requirement is that at any point in time the User A can have only one valid login. As in the samples below, the user may have attempted to login successfully multiple times, while his first session was still active. So, any logins that happened during the valid session needs to be flagged as duplicate. Example 1: In the first sample data below, while the user was still logged in from 00:12:38 to 01:00:02 (index 0), there is another login from the user at 00:55:14 to 01:00:02 (index 1). Similarly, if we compare index 2 and 3, we can see that the record at index 3 is duplicate login as per requirement.  start_time end_time 0 00:12:38 01:00:02 1 00:55:14 01:00:02 2 01:00:02 01:32:40 3 01:00:02 01:08:40 4 01:41:22 03:56:23 5 18:58:26 19:16:49 6 20:12:37 20:52:49 7 20:55:16 22:02:50 8 22:21:24 22:48:50 9 23:11:30 00:00:00  Expected output:  start_time end_time isDup 0 00:12:38 01:00:02 0 1 00:55:14 01:00:02 1 2 01:00:02 01:32:40 0 3 01:00:02 01:08:40 1 4 01:41:22 03:56:23 0 5 18:58:26 19:16:49 0 6 20:12:37 20:52:49 0 7 20:55:16 22:02:50 0 8 22:21:24 22:48:50 0 9 23:11:30 00:00:00 0  These duplicate records need to be updated to 1 at column isDup.  Example 2: Another sample of data as below. Here, while the user was still logged in between 13:36:10 and 13:50:16, there were 3 additional sessions too that needs to be flagged.  start_time end_time 0 13:32:54 13:32:55 1 13:36:10 13:50:16 2 13:37:54 13:38:14 3 13:46:38 13:46:45 4 13:48:59 13:49:05 5 13:50:16 13:50:20 6 14:03:39 14:03:49 7 15:36:20 15:36:20 8 15:46:47 15:46:47  Expected output:  start_time end_time isDup 0 13:32:54 13:32:55 0 1 13:36:10 13:50:16 0 2 13:37:54 13:38:14 1 3 13:46:38 13:46:45 1 4 13:48:59 13:49:05 1 5 13:50:16 13:50:20 0 6 14:03:39 14:03:49 0 7 15:36:20 15:36:20 0 8 15:46:47 15:46:47 0   What's the efficient way to compare the start time of the current record with previous records?

Answer: Map the time like values in columns start_time and end_time to pandas TimeDelta objects and subtract 1 seconds from the 00:00:00 timedelta values in end_time column. c = ['start_time', 'end_time'] s, e = df[c].astype(str).apply(pd.to_timedelta).to_numpy().T e[e == pd.Timedelta(0)] += pd.Timedelta(days=1, seconds=-1)  Then for each pair of start_time and end_time in the dataframe df mark the corresponding duplicate intervals using numpy broadcasting: m = (s[:, None] >= s) & (e[:, None] <= e) np.fill_diagonal(m, False) df['isDupe'] = (m.any(1) & ~df[c].duplicated(keep=False)).view('i1')   # example 1 start_time end_time isDupe 0 00:12:38 01:00:02 0 1 00:55:14 01:00:02 1 2 01:00:02 01:32:40 0 3 01:00:02 01:08:40 1 4 01:41:22 03:56:23 0 5 18:58:26 19:16:49 0 6 20:12:37 20:52:49 0 7 20:55:16 22:02:50 0 8 22:21:24 22:48:50 0 9 23:11:30 00:00:00 0 # example 2 start_time end_time isDupe 0 13:32:54 13:32:55 0 1 13:36:10 13:50:16 0 2 13:37:54 13:38:14 1 3 13:46:38 13:46:45 1 4 13:48:59 13:49:05 1 5 13:50:16 13:50:20 0 6 14:03:39 14:03:49 0 7 15:36:20 15:36:20 0 8 15:46:47 15:46:47 0