eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: My project has various private python packages developed internally in my organization. I am using [tool.poetry.source] to specify the PyPi server. I have a use case to specify custom PyPi server url for different packages. This is the content of my pyproject.toml [tool.poetry.dependencies] python = "^3.8" package-a = "0.1.2" package-b = "0.2.1" package-c = "0.4.2" [[tool.poetry.source]] name = "internal-repo-1" url = "https://<private-repo-1>" [[tool.poetry.source]] name = "internal-repo-2" url = "https://<private-repo-2>"  I want to use private-repo-1 from package-a and private-repo-2 for package-b and package-c. How can this be achieved ? Also can this be achieved without scanning all the private repositories for each and every package? I am using poetry for dependency management.

Answer: This is described in the docs: [tool.poetry.dependencies] python = "^3.8" package-a = { version = "0.1.2", source = "internal-repo-1" } package-b = { version = "0.2.1", source = "internal-repo-2" } package-c = { version = "0.4.2", source = "internal-repo-2" } [[tool.poetry.source]] name = "internal-repo-1" url = "https://<private-repo-1>" [[tool.poetry.source]] name = "internal-repo-2" url = "https://<private-repo-2>"
Question: Please I need some help with applying python UDF to run on my dbt models. I successfully created a python function in snowflake (DWH) and ran it against a table. This seems to work as expected, but implementing this on dbt seems to be a struggle. Some advice/help/direction will make my day. here is my python UDF created on snowflake create or replace function "077"."Unity".sha3_512(str varchar) returns varchar language python runtime_version = '3.8' handler = 'hash' as $$ import hashlib def hash(str): # create a sha3 hash object hash_sha3_512 = hashlib.new("sha3_512", str.encode()) return hash_sha3_512.hexdigest() $$ ;  The objective is the create the python function in dbt and apply it to the model below {{ config(materialized = 'view') }} WITH SEC AS( SELECT A."AccountID" AS AccountID, A."AccountName" AS AccountName , A."Password" AS Passwords, apply function here (A."Password") As SHash FROM {{ ref('Green', 'Account') }} A ) ----------------VIEW RECORD------------------------------ SELECT * FROM SEC  is there a way to do this please. Thank you

Answer: Assuming that UDF already exists in Snowflake: {{ config(materialized = 'view') }} WITH SEC AS( SELECT A."AccountID" AS AccountID, A."AccountName" AS AccountName , A."Password" AS Passwords, {{target.schema}}.sha3_512(A."Password") As SHash FROM {{ ref('Green', 'Account') }} A ) SELECT * FROM SEC;   The function could be created using on-run-start: on-run-start: - '{{ creating_udf()}}'  and macro: {% macro creating_udf() %} create function if not exists {{target.schema}}.sha3_512(str varchar) returns varchar language python runtime_version = '3.8' handler = 'hash' as $$ import hashlib def hash(str): # create a sha3 hash object hash_sha3_512 = hashlib.new("sha3_512", str.encode()) return hash_sha3_512.hexdigest() $$ ; {% endmacro %}
Question: I am trying to plot Precision Recall curve using PrecisionRecallDisplay from scikit-learn. I have model predicted values in y_pred and actual values in y_true. I can plot precision recall curve using the following syntax: metrics.PrecisionRecallDisplay.from_predictions(y_true, y_pred)  But I want to plot multiple curves (say by applying model on training or validation data) in the same plot. So is it possible to achieve this using PrecisionRecallDisplay? Or Is there some other standard way to achieve this using scikit-learn?

Answer: Since sklearn display routines are basically just matplotlib wrappers, the easiest way seems to be utilizing the ax argument, like this: import matplotlib.pyplot as plt fig, ax = plt.subplots() PrecisionRecallDisplay.from_predictions(y_train, y_pred_train, ax=ax) PrecisionRecallDisplay.from_predictions(y_test, y_pred, ax=ax) plt.show()
Question: I need to find a solution for the below problem in Python3. I tried itertools.combinations but not clear on how to do it. Prepare a 7-digit number that sums to 5. Each digit can be between 0-4 only. Also, there can be repetitions. Valid example numbers are - [ [2,1,1,0,0,1,0], [3,0,1,0,0,1,0], [0,0,0,4,0,0,1], [1,0,0,3,0,1,0], [1,1,1,1,0,1,0], ...... ]  As you can see, numbers may appear more than once in this list. How can I create a list of all combinations meeting the criteria above?

Answer: This function will find every combination, with repeated combinations, that sum to N: from itertools import product from typing import List, Tuple def perm_n_digit_total(n_digits, total, choices) -> List[Tuple]: return list(filter( lambda x: sum(x) == total, product(choices, repeat=n_digits) ))  Example: perm_n_digit_total(3, 1, range(4)) Out[43]: [(0, 0, 1), (0, 1, 0), (1, 0, 0)]  perm_n_digit_total(7, 5, range(4))[::50] Out[49]: [(0, 0, 0, 0, 0, 0, 5), (0, 0, 0, 3, 1, 1, 0), (0, 0, 2, 0, 3, 0, 0), (0, 1, 0, 1, 3, 0, 0), (0, 2, 0, 0, 1, 0, 2), (0, 4, 1, 0, 0, 0, 0), (1, 0, 1, 1, 1, 0, 1), (1, 1, 1, 1, 1, 0, 0), (2, 0, 1, 0, 0, 2, 0), (3, 1, 0, 0, 0, 1, 0)]
Question: I have a list of dictionaries, like this: [{'user': '123456', 'db': 'db1', 'size': '8628'} {'user': '123456', 'db': 'db1', 'size': '7168'} {'user': '123456', 'db': 'db1', 'size': '38160'} {'user': '222345', 'db': 'db3', 'size': '8628'} {'user': '222345', 'db': 'db3', 'size': '8628'} {'user': '222345', 'db': 'db5', 'size': '840'} {'user': '34521', 'db': 'db6', 'size': '12288'} {'user': '34521', 'db': 'db6', 'size': '476'} {'user': '2345156', 'db': 'db7', 'size': '5120'}.....]  This list contains millions of entries. Each user can be found in multiple dbs, each user can have multiple entires in the same db. I want to sum up how much is the size occupied by each user, per each db. I don't want to use pandas. At the moment I do it this way:  I create 2 lists of unique users and unique dbs Use those lists to iterate through the big list and sum up where user and db are the same  result = [] for user in unique_users: for db in unique_dbs: total_size = 0 for i in big_list: if (i['user'] == user and i['db'] == db): total_size += float(i['size']) if(total_size) > 0: row = {} row['user'] = user row['db'] = db row['size'] = total_size result.append(row)  The problem is that this triple for loop develops into something very large (hundreds of billions of iterations) which takes forever to sum up the result. If the big_list is small, this works very well. How should I approach this in order to keep it fast and simple? Thanks a lot!

Answer: There are two main issue with the current approach: the inefficient algorithm and the inefficient data structure. The first is that the algorithm used is clearly inefficient as it iterates many times over the big list. There is not need to iterate over the whole list to filter a unique user and db. You can iterate over the big list once and aggregate data using a dictionary. The key of the target dictionary is simply a (user, db) tuple. The value of the dictionary is total_size. Here is an untested example: # Aggregation part # Note: a default dict can be used instead to make the code possibly simpler aggregate_dict = dict() for i in big_list: key = (i['user'], i['db']) value = float(i['size']) if key in aggregate_dict: aggregate_dict[key] += value else: aggregate_dict[key] = value # Fast creation of `result` result = [] for user in unique_users: for db in unique_dbs: total_size = aggregate_dict.get((user, key)) if total_size is not None and total_size > 0: result.append({'user': user, 'db': db, 'size': total_size})  The other issue is the inefficient data structure: for each row, the keys are replicated while tuples can be used instead. In fact, a better data structure is to store a dictionary of (column, items) key-values where items is a list of items for the target column. This way of storing data is called a dataframe. This is roughly what Pandas uses internally (except it is a Numpy array which is even better as it is more compact and generally more efficient than a list for most operations). Using this data structure for both the input and the output should result in a significant speed up (if combined with Numpy) and a lower memory footprint.
Question: class User(BaseModel): name: str token: str fake_db = [ User(name='foo', token='a1'), User(name='bar', token='a2') ] async def get_user_by_token(token: str = Header()): for user in fake_db: if user.token == token: return user else: raise HTTPException(status_code=401, detail='Invalid token') @router.get(path='/test_a', summary='Test route A') async def test_route_a(user: User = Depends(get_user_by_token)): return {'name': user.name} @router.get(path='/test_b', summary='Test route B') async def test_route_a(user: User = Depends(get_user_by_token)): return {'name': user.name}  I would like to avoid code duplication. Is it possible to somehow set the line user: User = Depends(get_user_by_token) for the entire router? At the same time, I need the user object to be available in each method. It is very important that the openapi says that you need to specify a header with a token for the method. 

Answer: You can use the dependencies parameter to add global dependencies when creating the router instance: router = APIRouter(dependencies=[Depends(get_user_by_token)])  or, when adding the router to the app instance: app.include_router(router, dependencies=[Depends(get_user_by_token)])  Please have a look at FastAPI's documentation on Dependencies for more details. As for getting the return value of a global dependency, you can't really do that. The way around this issue is to store the returned value to request.state (as described here), which is used to store arbitrary state (see the implementation of State as well). Hence, you could have something like this: def get_user_by_token(request: Request, token: str = Header()): for user in fake_db: if user.token == token: request.state.user = user # ...  Then, inside your endpoint, you could retrieve the user object using request.state.user, as described in this answer.
Question: I have a list, L: L = ['top', 'left', 'behind', 'before', 'right', 'after', 'hand', 'side'] I have a pandas DataFrame, DF:     Text     the objects are both before and after the person   the object is behind the person   the object in right is next to top left hand side of person     I would like to extract all words in L from the DF column 'Text' in such a manner:     Text Extracted_Value     the objects are both before and after the person before_after   the object is behind the person behind   the object in right is next to top left hand side of person right_top left hand side     For case 1 and 2, my code is working: L = ['top', 'left', 'behind', 'before', 'right', 'after', 'hand', 'side'] pattern = r"(?:^|\s+)(" + "|".join(L) + r")(?:\s+|$)" df["Extracted_Value "] = ( df['Text'].str.findall(pattern).str.join("_").replace({"": None}) )  For CASE 3, I get right_top_hand. As in the third example, If identified words are contiguous, they are to be picked up as a phrase (one extraction). So in the object in right is next to top left hand side of person, there are two extractions - right and top left hand side. Hence, only these two extractions are separated by an _. I am not sure how to get it to work!

Answer: Try: df["Extracted_Value"] = ( df.Text.apply( lambda x: "|".join(w if w in L else "" for w in x.split()).strip("|") ) .replace(r"\|{2,}", "_", regex=True) .str.replace("|", " ", regex=False) ) print(df)  Prints:  Text Extracted_Value 0 the objects are both before and after the person before_after 1 the object is behind the person behind 2 the object in right is next to top left hand side of person right_top left hand side   EDIT: Adapting @Wiktor's answer to pandas: pattern = fr"\b((?:{'|'.join(L)})(?:\s+(?:{'|'.join(L)}))*)\b" df["Extracted_Value"] = ( df["Text"].str.extractall(pattern).groupby(level=0).agg("_".join) ) print(df)
Question: I want to achieve the following three things:  add arrows on x and y axes show only the values that are used add labels for coordinates  My code on this moment: x = [9, 8, 11, 11, 14, 13, 16, 14, 14] y = [9, 16, 15, 11, 10, 11, 10, 8, 8] fig = plt.figure(figsize=(7,7), dpi=300) axes = fig.add_axes([0,1,1,1]) axes.set_xlim(0, 17) axes.set_ylim(0, 17) axes.invert_yaxis() axes.scatter(x, y, color='green') axes.vlines(x, 0, y, linestyle="dashed", color='green') axes.hlines(y, 0, x, linestyle="dashed", color='green') axes.spines.right.set_visible(False) axes.spines.bottom.set_visible(False) plt.show()  Visually:  And plot that I want to realize 

Answer: You can draw arrows by overlaying triangle shaped points over the ends of your spines. You'll need to leverage some transforms, but you can also create your labels by manually adding text to your Axes objects as well. Labelling each coordinate can be done via axes.annotate, but you'll need to manually specify the location of each annotation to ensure they don't overlap with lines or other annotations. import matplotlib.pyplot as plt from matplotlib.ticker import FixedLocator x = [9, 8, 11, 11, 14, 13, 16, 14, 14] y = [9, 16, 15, 11, 10, 11, 10, 8, 8] fig = plt.figure(figsize=(7,7), dpi=300) axes = fig.add_axes([.05,.05,.9,.9]) # Plots the data axes.scatter(x, y, color='green') axes.vlines(x, 0, y, linestyle="dashed", color='green') axes.hlines(y, 0, x, linestyle="dashed", color='green') axes.set_xlim(0, 17) axes.set_ylim(0, 17) axes.set_xticks(x) axes.set_yticks(y) axes.invert_yaxis() # Move ticks to top side of plot axes.xaxis.set_tick_params( length=0, bottom=False, labelbottom=False, top=True, labeltop=True ) axes.xaxis.set_tick_params(length=0) # Add arrows to the spines by drawing triangle shaped points over them axes.plot(1, 1, '>k', transform=axes.transAxes, clip_on=False) axes.plot(0, 0, 'vk', transform=axes.transAxes, clip_on=False) axes.spines[['bottom', 'right']].set_visible(False) # Add labels for 0, F_1 and F_2 from matplotlib.transforms import offset_copy axes.text( 0, 1, s='0', fontstyle='italic', ha='right', va='bottom', transform=offset_copy(axes.transAxes, x=-5, y=5, fig=fig, units='points'), ) axes.text( 1, 1, s='$F_1$', fontstyle='italic', ha='right', va='bottom', transform=offset_copy(axes.transAxes, x=0, y=5, fig=fig, units='points'), ) axes.text( 0, 0, s='$F_2$', fontstyle='italic', ha='right', transform=offset_copy(axes.transAxes, x=-5, y=0, fig=fig, units='points'), ) # Add labels at each point. Leveraging the alignment of the text # AND padded offset. lc = ('top', 'center', 0, -5) ll = ('top', 'right', -5, -5) lr = ('top', 'left', 5, -5) ur = ('bottom', 'left', 5, 5) alignments = [lc, lc, lc, ll, lc, ll, lc, ur, lr] for i, (xc, yc, (va, ha, padx, pady)) in enumerate(zip(x, y, alignments)): axes.annotate( xy=(xc, yc), xytext=(padx, pady), text=f'$F(x_{i})$', ha=ha, va=va, textcoords='offset points') plt.show()
Question: I'm messing around with classes and data flow and I am having difficulties creating a list of classes inside the class (to give control of the list to the class in itself). class Person: listOfPeople = [] def __init__(self, name, age): self.name = name self.age = age self.listOfPeople = [] def set_age(self, age): if age <= 0: raise ValueError('The age must be positive') self._age = age def get_age(self): return self._age def AppendList(self): self.listOfPeople.append(self) def returnList(self): return self.listOfPeople age = property(fget=get_age, fset=set_age) john = Person('John', 18) barry = Person("Barry", 19) john.AppendList() barry.AppendList() print(Person.listOfPeople)  The output is simply  []  Let´s use this example. I want the class Person to have a list of people. That list of people has instances of the class it's in. I want the entire program to have access to this class, regardless of having an instance initialised. Is it even possible to do what I want in Python? My expected output is a list with the 2 instances I added to the list.

Answer: this code automatically adds new instances to Person.listOfPeople class Person: listOfPeople = [] def __init__(self, name, age): self.name = name self.age = age Person.listOfPeople.append(self) def set_age(self, age): if age <= 0: raise ValueError('The age must be positive') self._age = age def get_age(self): return self._age def AppendList(self): self.listOfPeople.append(self) def returnList(self): return self.listOfPeople def __repr__(self): #I've added __repr__ return self.name+' '+str(self.age) age = property(fget=get_age, fset=set_age) john = Person('John', 18) barry = Person("Barry", 19) # john.AppendList() # barry.AppendList() print(Person.listOfPeople)  the output: [John 18, Barry 19] is this what you need?
Question: I am trying to set a super-class field in a subclass using validator as follows: Approach 1 from typing import List from pydantic import BaseModel, validator, root_validator class ClassSuper(BaseModel): field1: int = 0 class ClassSub(ClassSuper): field2: List[int] @validator('field1') def validate_field1(cls, v, values): return len(values["field2"]) sub = ClassSub(field2=[1, 2, 3]) print(sub.field1) # It prints 0, but expected it to print 3  If I run the code above it prints 0, but I expected it to print 3 (which is basically len(field2)). However, if I use @root_validator() instead, I get the expected result. Approach 2 from typing import List from pydantic import BaseModel, validator, root_validator class ClassSuper(BaseModel): field1: int = 0 class ClassSub(ClassSuper): field2: List[int] @root_validator() def validate_field1(cls, values): values["field1"] = len(values["field2"]) return values sub = ClassSub(field2=[1, 2, 3]) print(sub.field1) # This prints 3, as expected  New to using pydantic and I am bit puzzled what I am doing wrong with the Approach 1. Thank you for your help.

Answer: The reason your Approach 1 does not work is because by default, validators for a field are not called, when the value for that field is not supplied (see docs). Your validate_field1 is never even called. If you add always=True to your @validator, the method is called, even if you don't provide a value for field1. However, if you try that, you'll see that it will still not work, but instead throw an error about the key "field2" not being present in values. This in turn is due to the fact that validators are called in the order they were defined. In this case, field1 is defined before field2, which means that field2 is not yet validated by the time validate_field1 is called. And values only contains previously-validated fields (see docs). Thus, at the time validate_field1 is called, values is simply an empty dictionary. Using the @root_validator is the correct approach here because it receives the entire model's data, regardless of whether or not field values were supplied explicitly or by default. And just as a side note: If you don't need to specify any parameters for it, you can use @root_validator without the parantheses. And as another side note: If you are using Python 3.9+, you can use the regular list class as the type annotation. (See standard generic alias types) That means field2: list[int] without the need for typing.List. Hope this helps.
Question: Let's assume I have a list, structured like this with approx 1 million elements: a = [["a","a"],["b","a"],["c","a"],["d","a"],["a","a"],["a","a"]]  What is the fastest way to remove all elements from a that have the same value at index 0? The result should be b = [["a","a"],["b","a"],["c","a"],["d","a"]]  Is there a faster way than this: processed = [] no_duplicates = [] for elem in a: if elem[0] not in processed: no_duplicates.append(elem) processed.append(elem[0])  This works but the appending operations take ages.

Answer: you can use set to keep the record of first element and check if for each sublist first element in this or not. it will took O(1) time compare to O(n) time to your solution to search. >>> a = [["a","a"],["b","a"],["c","a"],["d","a"],["a","a"],["a","a"]] >>> >>> seen = set() >>> new_a = [] >>> for i in a: ... if i[0] not in seen: ... new_a.append(i) ... seen.add(i[0]) ... >>> new_a [['a', 'a'], ['b', 'a'], ['c', 'a'], ['d', 'a']] >>>  Space complexity : O(N) Time complexity: O(N) Search if first element there or not : O(1) In case, no new list to be declared, then use del element, but this will increase time complexity
Question: I have the following dataframe (sample): import pandas as pd data = [['A', '2022-09-01'], ['A', '2022-09-03'], ['A', '2022-09-07'], ['A', '2022-09-08'], ['B', '2022-09-03'], ['B', '2022-09-07'], ['B', '2022-09-08'], ['B', '2022-09-09'], ['C', '2022-09-01'], ['C', '2022-09-03'], ['C', '2022-09-07'], ['C', '2022-09-10'], ['D', '2022-09-01'], ['D', '2022-09-03'], ['D', '2022-09-05'], ['D', '2022-09-07']] df = pd.DataFrame(data = data, columns = ['group', 'date']) group date 0 A 2022-09-01 1 A 2022-09-03 2 A 2022-09-07 3 A 2022-09-08 4 B 2022-09-03 5 B 2022-09-07 6 B 2022-09-08 7 B 2022-09-09 8 C 2022-09-01 9 C 2022-09-03 10 C 2022-09-07 11 C 2022-09-10 12 D 2022-09-01 13 D 2022-09-03 14 D 2022-09-05 15 D 2022-09-07  I would like to filter the dates which are available across all groups. For example, the date "2022-09-03" is available in groups: A, B, C and D so all groups. The date "2022-09-01" is only available in groups: A, C, and D which means it is missing in group B. Here is the desired output: data = [['A', '2022-09-03'], ['A', '2022-09-07'], ['B', '2022-09-03'], ['B', '2022-09-07'], ['C', '2022-09-03'], ['C', '2022-09-07'], ['D', '2022-09-03'], ['D', '2022-09-07']] df_desired = pd.DataFrame(data = data, columns = ['group', 'date']) group date 0 A 2022-09-03 1 A 2022-09-07 2 B 2022-09-03 3 B 2022-09-07 4 C 2022-09-03 5 C 2022-09-07 6 D 2022-09-03 7 D 2022-09-07  I know how to filter groups with all the same values within a group, but I want to filter the dates which are available in each group. So I was wondering if anyone knows how to perform this using pandas?

Answer: You can use set operations: # which dates are common to all groups? keep = set.intersection(*df.groupby('group')['date'].agg(set)) # {'2022-09-03', '2022-09-07'} # keep only the matching ones out = df[df['date'].isin(keep)]  output:  group date 1 A 2022-09-03 2 A 2022-09-07 4 B 2022-09-03 5 B 2022-09-07 9 C 2022-09-03 10 C 2022-09-07 13 D 2022-09-03 15 D 2022-09-07  comparison of approaches: # set operations 669 µs ± 13.4 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each) # nunique 750 µs ± 16.3 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each) # 2D reshaping (crosstab) 5.45 ms ± 418 µs per loop (mean ± std. dev. of 7 runs, 100 loops each) # on 200k rows (random or like original) # set operations 21.1 ms ± 2.23 ms per loop (mean ± std. dev. of 7 runs, 10 loops each) # nunique 26.7 ms ± 1.48 ms per loop (mean ± std. dev. of 7 runs, 10 loops each) # crosstab 47.8 ms ± 3.69 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)