eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I have a sqlite.db file which has 5 columns and 10million rows. I have created a api using fastapi, now in one of the api methods I want to connect to that sqlite.db file and fetch content based on certain conditions (based on the columns present). I mostly will be using SELECT and WHERE. How can I do it by also taking advantage of async requests. I have came across Tortoise ORM but I am not sure how to properly use it to fetch results. from fastapi import FastAPI, UploadFile, File, Form from fastapi.middleware.cors import CORSMiddleware DATABASE_URL = "sqlite:///test.db" @app.post("/test") async def fetch_data(id: int): query = "SELECT * FROM tablename WHERE ID={}".format(str(id)) # how can I fetch such query faster from 10 million records while taking advantage of async func return results 

Answer: You are missing a point here, defining a function with async is not enough. You need to use an asynchronous Database Driver to taking the advantage of using a coroutine. Encode's Databases library is great for this purpose. pip install databases  You can also install the required database drivers with: pip install databases[sqlite]  In your case, this should do good. from fastapi import FastAPI, UploadFile, File, Form from fastapi.middleware.cors import CORSMiddleware from databases import Database database = Database("sqlite:///test.db") @app.on_event("startup") async def database_connect(): await database.connect() @app.on_event("shutdown") async def database_disconnect(): await database.disconnect() @app.post("/test") async def fetch_data(id: int): query = "SELECT * FROM tablename WHERE ID={}".format(str(id)) results = await database.fetch_all(query=query) return results
Question: I want to keep my type aliases in one module, say my_types, to be able to use them anywhere in my application (similar to the standard typing module). But mypy complains that the forward reference to class X is not defined. If I define class X later in that same module, it’s okay, but if it defined in another one, mypy gets upset. So my question is, how do I keep all my type aliases in one module without mypy producing an error about forward references that are not defined in the same module? Or is that a wrong approach somehow? Here is my example code: from my_types import SomeXs class X: pass  Type aliases are defined like so: # my_types.py from typing import List SomeXs = List['X']  When I run mypy, I get an error that X is not defined: $ mypy module.py my_types.py:4: error: Name 'X' is not defined Found 1 error in 1 file (checked 1 source file) 

Answer: I’m gonna share the solution I found in the mypy documentation common issues section here: It is necessary for mypy to have access to the definition of X. To avoid an import cycle, the mypy documentation recommends a trick - only import the definition that would create a cyclic import when type checking. It goes like this: from typing import List, TYPE_CHECKING if TYPE_CHECKING: from main import X SomeXs = List['X']  This makes the Python interpreter ignore the import when executing this code, but mypy still uses the definition of X from main. Et voilá: $ mypy *.py Success: no issues found in 2 source files
Question: I have 2 lists similar to these: l1 = [('zero', 0),('one', 2),('two', 3),('three', 3),('four', 5)] l2 = [('zero', 0),('one', 3),('four', 2),('ten', 3),('twelve', 8)]  I want to compare the lists and  remove duplicates from both if both values are the same if the first value is a match remove the tuple from the list where the second value is lower  I can do the first with l3 = [(a,b) for (a,b) in l1 if (a,b) not in l2] l4 = [(a,b) for (a,b) in l2 if (a,b) not in l1]  or using set though it doesn't preserve order l3 = set(l1) - set(l2)  but I'm having a hard time figuring out the second. I tried to start with removing based on just the first value with l3 = [(a,b) for (a,b) in l1 if a not in l2]  but that doesn't work. My desired output for l3 & l4 is: l3 [('two', 3),('three', 3),('four', 5)] l4 [('one', 3),('ten', 3),('twelve', 8)]  Any guidance would be appreciated.

Answer: You could do: d1 = dict(l1) d2 = dict(l2) l3 = [(k, v) for k, v in d1.items() if k not in d2 or d2[k] < v] l4 = [(k, v) for k, v in d2.items() if k not in d1 or d1[k] < v] print(l3) print(l4)  Output [('two', 3), ('three', 3), ('four', 5)] [('one', 3), ('ten', 3), ('twelve', 8)]  The idea is to use dictionaries for fast lookups of matching first values, if any, and then check if the corresponding second value is less to the one in the current list.
Question: I am struggling to figure out an efficient algorithm to perform the following task: Given two arrays A and B with equal length, the difference between the two arrays is defined as: diff = |a[0]-b[0]| + |a[1]-b[1]| +...+|a[a.length-1]-b[b.length-1]|  I am required to find the minimum possible difference between A and B, and I am allowed to replace at most one element from A with any other element in A. Note that you are not required to perform a replacement. For example: A = [1,3,5] B = [5,3,1]  If we replace A[2] with A[0], then the difference between the two arrays is: |1-5| + |3-3| + |1-1| = 4  This is the minimal possible difference between the two arrays. No other replacement of an element in A with another element in A would result in a smaller difference between A and B. How would I go about solving this problem? I know how to solved the problem in O(n^2) (brute force), but struggling to figure out a more efficient way. Thanks!

Answer: I'll implement Shridhar's suggestion of identifying the best modification for each element individually in O(n log n) time and taking the best one. import bisect def abs_diff(x, y): return abs(x - y) def find_nearest(sorted_a, y): i = bisect.bisect(sorted_a, y) return min( sorted_a[max(i - 1, 0) : min(i + 1, len(sorted_a))], key=lambda z: abs_diff(z, y), ) def improvement(x, y, z): return abs_diff(x, y) - abs_diff(z, y) def min_diff(a, b): sorted_a = sorted(a) nearest = [find_nearest(sorted_a, y) for y in b] return sum(map(abs_diff, a, b)) - max(map(improvement, a, b, nearest)) print(min_diff([1, 3, 5], [5, 3, 1]))
Question: I am working with Text Classification problem where I want to use the BERT model as the base followed by Dense layers. I want to know how does the 3 arguments work? For example, if I have 3 sentences as: 'My name is slim shade and I am an aspiring AI Engineer', 'I am an aspiring AI Engineer', 'My name is Slim'  SO what will these 3 arguments do? What I think is as follows:  max_length=5 will keep all the sentences as of length 5 strictly padding=max_length will add a padding of 1 to the third sentence truncate=True will truncate the first and second sentence so that their length will be strictly 5.  Please correct me if I am wrong. Below is my code which I have used. ! pip install transformers==3.5.1 from transformers import BertTokenizerFast tokenizer = BertTokenizerFast.from_pretrained('bert-base-uncased') tokens = tokenizer.batch_encode_plus(text,max_length=5,padding='max_length', truncation=True) text_seq = torch.tensor(tokens['input_ids']) text_mask = torch.tensor(tokens['attention_mask']) 

Answer: What you have assumed is almost correct, however, there are few differences. max_length=5, the max_length specifies the length of the tokenized text. By default, BERT performs word-piece tokenization. For example the word "playing" can be split into "play" and "##ing" (This may not be very precise, but just to help you understand about word-piece tokenization), followed by adding [CLS] token at the beginning of the sentence, and [SEP] token at the end of sentence. Thus, it first tokenizes the sentence, truncates it to max_length-2 (if truncation=True), then prepend [CLS] at the beginning and [SEP] token at the end.(So a total length of max_length) padding='max_length', In this example it is not very evident that the 3rd example will be padded, as the length exceeds 5 after appending [CLS] and [SEP] tokens. However, if you have a max_length of 10. The tokenized text corresponds to [101, 2026, 2171, 2003, 11754, 102, 0, 0, 0, 0], where 101 is id of [CLS] and 102 is id of [SEP] tokens. Thus, padded by zeros to make all the text to the length of max_length Likewise, truncate=True will ensure that the max_length is strictly adhered, i.e, longer sentences are truncated to max_length only if truncate=True
Question: Consider this contrived example; @dataclass class A: name: str = "John" .... @dataclass class B: name: str = "Doe"  Q: How do I type hint an object that has an attribute, such as the following? def print_name(obj: HasAttr['name']) print(obj.name)  I understand the SO rule on showing what you have tried. The best I can offer is that I've searched the docs; Pep526, PythonSheets, Docs, and am aware of this SO Question. None seem to help (or maybe I missed it.) [Edit] I recognize that you can get there with inheritance, but I don't want to go that route.

Answer: So, what you are describing is structural typing. This is distinct from the class-based nominal subtyping that the python typing system is based on. However, structural subtyping is sort of the statically typed version of Python's dynamic duck typing. Python's typing system allows a form of this through typing.Protocol. An example, suppose we have a Python module, test_typing.py: from typing import Protocol from dataclasses import dataclass class Named(Protocol): name: str @dataclass class A: name: str id: int @dataclass class B: name: int @dataclass class C: foo: str def frobnicate(obj: Named) -> int: return sum(map(ord, obj.name)) frobnicate(A('Juan', 1)) frobnicate(B(8)) frobnicate(C('Jon'))  Using mypy version 0.790: (py38) juanarrivillaga@Juan-Arrivillaga-MacBook-Pro ~ % mypy test_typing.py test_typing.py:28: error: Argument 1 to "frobnicate" has incompatible type "B"; expected "Named" test_typing.py:28: note: Following member(s) of "B" have conflicts: test_typing.py:28: note: name: expected "str", got "int" test_typing.py:29: error: Argument 1 to "frobnicate" has incompatible type "C"; expected "Named" Found 2 errors in 1 file (checked 1 source file)
Question: In Django: I have created a super user and can view all the users I have also implemented forgot password for my user, who can input their email and a password reset link is sent to their email and then the user can reset his password But how can admin change some users password from the admin dashboard

Answer: This answer is just an extension of answer by @kunal Sharma To change user password from Django admin Go into the user  and click this form, and a form below will be shown, change password there
Question: Given the following chart created in plotly.  I want to add the percentage values of each count for M and F categories inside each block. The code used to generate this plot. arr = np.array([ ['Dog', 'M'], ['Dog', 'M'], ['Dog', 'F'], ['Dog', 'F'], ['Cat', 'F'], ['Cat', 'F'], ['Cat', 'F'], ['Cat', 'M'], ['Fox', 'M'], ['Fox', 'M'], ['Fox', 'M'], ['Fox', 'F'], ['Dog', 'F'], ['Dog', 'F'], ['Cat', 'F'], ['Dog', 'M'] ]) df = pd.DataFrame(arr, columns=['A', 'G']) fig = px.histogram(df, x="A", color='G', barmode="stack") fig.update_layout(height=400, width=800) fig.show() 

Answer: As far as I know histograms in Plotly don't have a text attribute. But you could generate the bar chart yourself and then add the percentage via the text attribute. import numpy as np import pandas as pd import plotly.express as px arr = np.array([ ['Dog', 'M'], ['Dog', 'M'], ['Dog', 'F'], ['Dog', 'F'], ['Cat', 'F'], ['Cat', 'F'], ['Cat', 'F'], ['Cat', 'M'], ['Fox', 'M'], ['Fox', 'M'], ['Fox', 'M'], ['Fox', 'F'], ['Dog', 'F'], ['Dog', 'F'], ['Cat', 'F'], ['Dog', 'M'] ]) df = pd.DataFrame(arr, columns=['A', 'G']) df_g = df.groupby(['A', 'G']).size().reset_index() df_g['percentage'] = df.groupby(['A', 'G']).size().groupby(level=0).apply(lambda x: 100 * x / float(x.sum())).values df_g.columns = ['A', 'G', 'Counts', 'Percentage'] px.bar(df_g, x='A', y=['Counts'], color='G', text=df_g['Percentage'].apply(lambda x: '{0:1.2f}%'.format(x)))
Question: Suppose I have a numpy array x of shape [1,5]. I want to expand it along axis 0 such that the resulting array y has shape [10,5] and y[i:i+1,:] is equal to x for each i. If x were a pytorch tensor I could simply do y = x.expand(10,-1)  But there is no expand in numpy and the ones that look like it (expand_dims and repeat) don't seem to behave like it.  Example: >>> import torch >>> x = torch.randn(1,5) >>> print(x) tensor([[ 1.3306, 0.0627, 0.5585, -1.3128, -1.4724]]) >>> print(x.expand(10,-1)) tensor([[ 1.3306, 0.0627, 0.5585, -1.3128, -1.4724], [ 1.3306, 0.0627, 0.5585, -1.3128, -1.4724], [ 1.3306, 0.0627, 0.5585, -1.3128, -1.4724], [ 1.3306, 0.0627, 0.5585, -1.3128, -1.4724], [ 1.3306, 0.0627, 0.5585, -1.3128, -1.4724], [ 1.3306, 0.0627, 0.5585, -1.3128, -1.4724], [ 1.3306, 0.0627, 0.5585, -1.3128, -1.4724], [ 1.3306, 0.0627, 0.5585, -1.3128, -1.4724], [ 1.3306, 0.0627, 0.5585, -1.3128, -1.4724], [ 1.3306, 0.0627, 0.5585, -1.3128, -1.4724]]) 

Answer: You can achieve that with np.broadcast_to. But you can't use negative numbers: >>> import numpy as np >>> x = np.array([[ 1.3306, 0.0627, 0.5585, -1.3128, -1.4724]]) >>> print(np.broadcast_to(x,(10,5))) [[ 1.3306 0.0627 0.5585 -1.3128 -1.4724] [ 1.3306 0.0627 0.5585 -1.3128 -1.4724] [ 1.3306 0.0627 0.5585 -1.3128 -1.4724] [ 1.3306 0.0627 0.5585 -1.3128 -1.4724] [ 1.3306 0.0627 0.5585 -1.3128 -1.4724] [ 1.3306 0.0627 0.5585 -1.3128 -1.4724] [ 1.3306 0.0627 0.5585 -1.3128 -1.4724] [ 1.3306 0.0627 0.5585 -1.3128 -1.4724] [ 1.3306 0.0627 0.5585 -1.3128 -1.4724] [ 1.3306 0.0627 0.5585 -1.3128 -1.4724]]
Question: My orchestrator receives a payload, with that payload it contains instructions that need to be passed along with other sets of data to activity functions. how do I pass multiple parameters to an activity function? Or do I have to mash all my data together? def orchestrator_function(context: df.DurableOrchestrationContext): # User defined configuration instructions: str = context.get_input() task_batch = yield context.call_activity("get_tasks", None) # Need to pass in instructions too parallel_tasks = [context.call_activity("perform_task", task) for task in task_batch] results = yield context.task_all(parallel_tasks) return results  The perform_task activity needs both the items from task_batch and the user input instructions Do I do something in my function.json? Workaround Not ideal, but I can pass multiple parameters as a single Tuple something = yield context.call_activity("activity", ("param_1", "param_2"))  I then just need to reference the correct index of the parameter in the activity.

Answer: Seems there's no text-book way to do it. I have opted to give my single parameter a generic name like parameter or payload. Then when passing in the value in the orchestrator I do it like so: payload = {"value_1": some_var, "value_2": another_var} something = yield context.call_activity("activity", payload)  then within the activity function, I unpack it again. edit: Some buried documentation seems to show that https://learn.microsoft.com/en-us/azure/azure-functions/durable/durable-functions-error-handling?tabs=python
Question: I am new to geopandas and would like to plot only the outline of a polygon, similar to the function ST_Boundary() in PostGIS I have a geodataframe states containing polygons for each state states = counties.dissolve(by='STATEFP')  When I subset by one state, I am able to plot that state: states.loc[states.index.isin(['06'])]['geometry']  I am only interested in the outline but it is not clear in the documentation how to convert a polygon to line geometry however. Is there a useful method in geopandas or another spatial library that might help in converting a polygon to a linestring? 

Answer: You can get boundary as states.boundary  Alternatively, if you want only exterior boundary you get it as states.exterior  Those give you new GeoSeries with line geometry.
Question: I'm trying to deploy a Cloud Function and it is returning me all the time the following error after importing bigquery from google.cloud ImportError: cannot import name 'bigquery' from 'google.cloud' (unknown location)  I've tried to install all the newest versions and remove and reinstall and persists Any idea?

Answer: Try adding something like google-cloud-bigquery==2.3.1 into requirements.txt of your Google Cloud Function
Question: I use FastAPi and Pydantic to model the requests and responses to an POST API. I defined three class: from pydantic import BaseModel, Field from typing import List, Optional, Dict class RolesSchema(BaseModel): roles_id: List[str] class HRSchema(BaseModel): pk: int user_id: str worker_id: str worker_name: str worker_email: str schedulable: bool roles: RolesSchema state: dict class CreateHR(BaseModel): user_id: str worker_id: str worker_name: str worker_email: str schedulable: bool roles: RolesSchema  And My API's program: @router.post("/humanResource", response_model=HRSchema) async def create_humanResource(create: CreateHR): query = HumanResourceModel.insert().values( user_id=create.user_id, worker_id=create.worker_id, worker_name=create.worker_name, worker_email=create.worker_email, schedulable=create.schedulable, roles=create.roles ) last_record_id = await database.execute(query) return {"status": "Successfully Created!"}  Input data format is json: { "user_id": "123", "worker_id": "010", "worker_name": "Amos", "worker_email": "Amos@mail.com", "schedulable": true, "roles": {"roles_id": ["001"]} }  When I executed, I got TypeError: Object of type RolesSchema is not JSON serializable. How can I fix the program to normal operation?

Answer: Try to use roles=create.roles.dict() for creating query instead of roles=create.roles
Question: Given a dataset like this: import pandas as pd rows = [{'key': 'ABC', 'freq': 100}, {'key': 'DEF', 'freq': 60}, {'key': 'GHI', 'freq': 50}, {'key': 'JKL', 'freq': 40}, {'key': 'MNO', 'freq': 13}, {'key': 'PQR', 'freq': 11}, {'key': 'STU', 'freq': 10}, {'key': 'VWX', 'freq': 10}, {'key': 'YZZ', 'freq': 3}, {'key': 'WHYQ', 'freq': 3}, {'key': 'HOWEE', 'freq': 2}, {'key': 'DUH', 'freq': 1}, {'key': 'HAHA', 'freq': 1}] df = pd.DataFrame(rows) df['percent'] = df['freq'] / sum(df['freq'])  [out]: key freq percent 0 ABC 100 0.328947 1 DEF 60 0.197368 2 GHI 50 0.164474 3 JKL 40 0.131579 4 MNO 13 0.042763 5 PQR 11 0.036184 6 STU 10 0.032895 7 VWX 10 0.032895 8 YZZ 3 0.009868 9 WHYQ 3 0.009868 10 HOWEE 2 0.006579 11 DUH 1 0.003289 12 HAHA 1 0.003289  The goal is to  select 1 example from the top 50-100 percentile of the frequency select 2 examples from the 10-50 percentile and select 4 example from < 10 percentile  In this case, the answer that fits are:  Pick 1 from ['ABC', 'DEF'] Pick 2 from ['GHI', 'JKL', 'MNO', 'PQR'] Pick 4 from ['VWX', 'STU', 'YZZ', 'WHYQ', 'HOWEE', 'HAHA', 'DUH']  I've tried this: import random import pandas as pd rows = [{'key': 'ABC', 'freq': 100}, {'key': 'DEF', 'freq': 60}, {'key': 'GHI', 'freq': 50}, {'key': 'JKL', 'freq': 40}, {'key': 'MNO', 'freq': 13}, {'key': 'PQR', 'freq': 11}, {'key': 'STU', 'freq': 10}, {'key': 'VWX', 'freq': 10}, {'key': 'YZZ', 'freq': 3}, {'key': 'WHYQ', 'freq': 3}, {'key': 'HOWEE', 'freq': 2}, {'key': 'DUH', 'freq': 1}, {'key': 'HAHA', 'freq': 1}] df = pd.DataFrame(rows) df['percent'] = df['freq'] / sum(df['freq']) bin_50_100 = [] bin_10_50 = [] bin_10 = [] total_percent = 1.0 for idx, row in df.sort_values(by=['freq', 'key'], ascending=False).iterrows(): if total_percent > 0.5: bin_50_100.append(row['key']) elif 0.1 < total_percent < 0.5: bin_10_50.append(row['key']) else: bin_10.append(row['key']) total_percent -= row['percent'] print(random.sample(bin_50_100, 1)) print(random.sample(bin_10_50, 2)) print(random.sample(bin_10, 4))  [out]: ['DEF'] ['MNO', 'PQR'] ['HOWEE', 'WHYQ', 'HAHA', 'DUH']  But is there a simpler way to solve the problem?

Answer: Let's try: bins = [0, 0.1, 0.5, 1] samples = [3,3,1] df['sample'] = pd.cut(df.percent[::-1].cumsum(), # accumulate percentage bins=[0, 0.1, 0.5, 1], # bins labels=False # num samples ).astype(int) df.groupby('sample').apply(lambda x: x.sample(n=samples[x['sample'].iloc[0])] )  Output:  key freq percent sample sample 1 0 ABC 100 0.328947 1 2 2 GHI 50 0.164474 2 5 PQR 11 0.036184 2 4 7 VWX 10 0.032895 4 6 STU 10 0.032895 4 12 HAHA 1 0.003289 4 10 HOWEE 2 0.006579 4
Question: How to run a server in python? I already have tried: python -m SimpleHTTPServer python -m HTTPServer  but its says to me: invalid syntax  Can someone help me? Thanks!

Answer: You can use this command in cmd or terminal python -m SimpleHTTPServer <port_number> # Python 2.x  Python 3.x python3 -m http.server # Python 3x  By default, this will run the contents of the directory on a local web server, on port 8000. You can go to this server by going to the URL localhost:8000 in your web browser.
Question: Given the below, how can i get the animal, age and gender into each of the table cells please? Currently all the data ends up in one cell. Thanks from rich.console import Console from rich.table import Table list = [['Cat', '7', 'Female'], ['Dog', '0.5', 'Male'], ['Guinea Pig', '5', 'Male']] table1 = Table(show_header=True, header_style='bold') table1.add_column('Animal') table1.add_column('Age') table1.add_column('Gender') for row in zip(*list): table1.add_row(' '.join(row)) console.print(table1) 

Answer: Just use * to unpack the tuple and it should work fine. for row in zip(*list): table1.add_row(*row)  Note that table1.add_row(*('Cat', 'Dog', 'Guinea Pig'))  is equivalent to table1.add_row('Cat', 'Dog', 'Guinea Pig')  While previously your approach was equivalent to table1.add_row('Cat Dog Guinea Pig')
Question: When I am trying to import a metric from sklearn, I get the following error: from sklearn.metrics import mean_absolute_percentage_error ImportError: cannot import name 'mean_absolute_percentage_error' from 'sklearn.metrics' /Users/carter/opt/anaconda3/lib/python3.8/site-packages/sklearn/metrics/__init__.py)  I have used conda update all, and reinstalled scikit-learn to no avail. Any other reasons this might happen and solutions?

Answer: The function mean_absolute_percentage_error is new in scikit-learn version 0.24 as noted in the documentation. As of December 2020, the latest version of scikit-learn available from Anaconda is v0.23.2, so that's why you're not able to import mean_absolute_percentage_error. You could try installing the latest version from source instead, or implement the function you need yourself. The source is available here if you'd like to take a look.
Question: When assigning an incorrect attribute to a Pydantic model field, no validation error occurs. from pydantic import BaseModel class pyUser(BaseModel): username: str class Config: validate_all = True validate_assignment = True person = pyUser(username=1234) person.username >>>1234 try_again = pyUser() pydantic.error_wrappers.ValidationError: [ErrorWrapper(exc=MissingError(), loc=('username',))] <class '__main__.pyUser'>  How can I get pydantic to validate on assignment?

Answer: It is expected behaviour according to the documentation:  str strings are accepted as-is, int, float and Decimal are coerced using str(v)  You can use the StrictStr, StrictInt, StrictFloat, and StrictBool types to prevent coercion from compatible types. from pydantic import BaseModel, StrictStr class pyUser(BaseModel): username: StrictStr class Config: validate_all = True validate_assignment = True person = pyUser(username=1234) # ValidationError `str type expected` print(person.username)