eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: So this question is specifically about querying a timezone aware date range with max min in Django 4.2. Timezone is set as TIME_ZONE = 'UTC' in settings.py and the model in question has two fields: open_to_readers = models.DateTimeField(auto_now=False, verbose_name="Campaign Opens") close_to_readers = models.DateTimeField(auto_now=False, verbose_name="Campaign Closes")  The query looks like allcampaigns = Campaigns.objects.filter(open_to_readers__lte=today_min, close_to_readers__gt=today_max)  Failed Solution 1  today_min = datetime.combine(timezone.now().date(), datetime.today().time().min) today_max = datetime.combine(timezone.now().date(), datetime.today().time().max) print("Today Min", today_min, " & Today Max", today_max)  returns the following which would be a suitable date range except it also gives the error below for both min and max. Today Min 2023-10-21 00:00:00 & Today Max 2023-10-21 23:59:59.999999 DateTimeField ... received a naive datetime (9999-12-31 23:59:59.999999) while time zone support is active.  Partial Working Solution today_min = timezone.now() allcampaigns = Campaigns.objects.filter(open_to_readers__lte=today_min, close_to_readers__gt=today_min)  Returns results without error but the time given is the current time and not the minimum or maximum for the day. Failed Solution 2 from here:  now = datetime.now(timezone.get_default_timezone()) today_min = now.min today_max = now.max print("Today Min", today_min, " & Today Max", today_max)  Returns Today Min 0001-01-01 00:00:00 & Today Max 9999-12-31 23:59:59.999999 and the aforementioned timezone error. How can I create two timezone aware datetime for the minumum and maximum parts of the day?

Answer: Likely the easiest way is just: from datetime import timedelta from django.utils.timezone import now date_min = now().replace(hour=0, minute=0, second=0, microsecond=0) date_max = date_min + timedelta(days=1) - date_min.resolution Campaigns.objects.filter( open_to_readers__lte=today_min, close_to_readers__gt=today_min )
Question: This site page has tooltips appearing when hovering over values in columns "Score" and "XP LVL". It appears that read_html will concatenate cell content and tooltip. Splitting those in post-processing is not always obvious and I seek a way to have read_html handle them separately, possibly return them as two columns. This is how the first row appears online:     (Rank)# Name Score XP LVL Victories / Total Victory Ratio     1 Rainin☆☆☆☆ 6129 447 408 / 531 76%      where "Score"'s "6129" carries tooltip "Max6129" where, more annoyingly, "XP LVL"'s "447" carries tooltip "21173534 pts"  This is how it appears after reading: pd.read_html('https://stats.gladiabots.com/pantheon?', header=0, flavor="html5lib")[0] # Name Score XP LVL Victories / Total \ 0 1 Rainin☆☆☆☆ 6129Max 6129 44721173534 pts 408 / 531  See "44721173534 pts" is the concatenation of "447" and "21173534 pts". "XP LVL" values have a variable number of digits, so splitting the string in the post-processing phase would require being pretty smart about it and I woud like to explore the "let read_html do the split", first. (The special flavor="html5lib" was added because the page is dynamically-generated) I have not found any mention of tooltips in the docs

Answer: It turns out that this is because pandas uses the .text attribute of the <td> bs4.element.Tag objects and this one concatenate (without any separator) the texts of all the tag's children. In the first row of the table, the score has two children 6129 and Max 6129, thus the concat. <td nowrap="" class="barContainer"> <div class="scoreBar" style="width: 100%;"></div> <div class="maxScoreBar" style="width: 0%;"></div> <span class="barLabel tooltipable"> "6129" <span class="tooltip"> "Max 6129" </span> </span> </td>  A quick/hacky solution would be to override the _text_getter method of the parser used by pandas and replace .text with get_text that has a separator parameter : def _text_getter(self, obj): return obj.get_text(separator="_", strip=True) # I choosed "_" pd.io.html._BeautifulSoupHtml5LibFrameParser._text_getter = _text_getter  With this modification, read_html gives this df :  # Name Score XP LVL Victories / Total Victory_Ratio 0 1 Rainin☆☆☆☆ 6129_Max 6129 447_21173534 pts 408 / 531 76% 1 2 ZM_XL☆☆☆ 5888_Max 6025 344_15942978 pts 3685 / 6748 54% 2 3 UzuraGames☆ 5555_Max 5586 119_4688941 pts 610 / 1109 55% .. ... ... ... ... ... ... 997 998 Tekuma 3183_Max 3460 27_370585 pts 151 / 304 49% 998 999 hemi 3183_Max 3227 10_49432 pts 29 / 62 46% 999 1000 wanna bet kid? 3183_Max 3304 13_85777 pts 51 / 95 53% [1000 rows x 6 columns]  And this way, you can extract / disattach the values of the two concerned columns : scores = df.pop("Score").str.extract(r"(?P<Score>\d+)_Max (?P<Max>\d+)") xplvls = df.pop("XP LVL").str.extract(r"(?P<XPLVL>\d+)_(?P<PTS>\d+)") out = pd.concat([df, scores, xplvls], axis=1)  Output : print(out) # with only `scores` and `xplvls` Score Max XPLVL PTS 0 6129 6129 447 21173534 1 5888 6025 344 15942978 2 5555 5586 119 4688941 .. ... ... ... ... 997 3183 3460 27 370585 998 3183 3227 10 49432 999 3183 3304 13 85777 [1000 rows x 4 columns]
Question: I have the following code, which works as expected on 3.9 and 3.10: from enum import Enum class Environment(Enum): DEV = () INT = () PROD = () def __init__(self): self._value_ = self.name.lower() def __str__(self): return self.name def get_url(self): if self is Environment.DEV: return 'http://localhost' else: return f'https://{self.value}.my.domain'  Here is the result in a REPL: >>> Environment.DEV.get_url() 'http://localhost' >>> Environment.INT.get_url() 'https://int.my.domain' >>> Environment.PROD.get_url() 'https://prod.my.domain'  I did this because I read that defining an __init__ method for enums allows passing tuples when assigning values, and I wanted to avoid the redundant/error prone explicit declarations such as:  DEV = 'DEV' INT = 'INT' PROD = 'PROD'  Now, when I run the same code on Python 3.11 or 3.12, every call to get_url returns the localhost URL. Trying it out in the REPL, we get: >>> Environment.DEV.get_url() 'http://localhost' >>> Environment.INT.get_url() 'http://localhost' >>> Environment.PROD.get_url() 'http://localhost'  I even noticed that all of the enum values are equal to the first one: >>> Environment.DEV <Environment.DEV: 'dev'> >>> Environment.INT <Environment.DEV: 'dev'> >>> Environment.PROD <Environment.DEV: 'dev'>  Has something changed in python 3.11 which makes using empty tuples at assignation forbidden/broken? How could I get the same result in those versions, i.e. declare the enum values without having to explicitly pass any argument to __init__?

Answer: You could use auto() and _generate_next_value_: from enum import Enum, auto class Environment(Enum): def _generate_next_value_(name, start, count, last_values): return name DEV = auto() INT = auto() PROD = auto()  This will automatically generate values based on the names. Note that while _generate_next_value_ is currently documented as a staticmethod, applying the staticmethod decorator is optional, and didn't work on previous versions - previous versions required a bare function.
Question: I am trying to replace a value in a pandas dataframe based on another column value. I made sample code below to replicate the issue, but essentially I want to add a column to an existing dataframe and then replace placeholder information based on another column's value. The dataframe I am using (not in the example) is based on an excel document that will be used to webscrape information that I want returned in the new column based on the other column's value. Just wanted to mention that before people asked why not just start the ex_list data in the dataframe. Additionally I only want it replace where the condition is met, not replacing the entire column with a set value. Sample Code ## this would be the excel document df sample_df = pd.DataFrame({"a":[1,2,3,4,5]}) sample_df["b"] = "" ## this data would be webscrapped using information above ex_list = [[1, "CHANGE"],[4, "CHANGE"]] for sub in ex_list: location = sample_df.loc[sample_df['a']==sub[0], 'b'].iloc[0] sample_df.replace(location, sub[1]) sample.head()  I also tried this just as a quick playaround but it produced the same output sample_df = pd.DataFrame({"a":[1,2,3,4,5]}) sample_df["b"] = "" ex_list = [[1, "CHANGE"],[4, "CHANGE"]] for sub in ex_list: sample_df[sample_df['a']==sub[0], 'b'].iloc[0] += sub[1] sample_df.head()  Both outputs are the same and show no change:  a b 0 1 1 2 2 3 3 4 4 5  The output I am hoping for is this  a b 0 1 CHANGE 1 2 2 3 3 4 CHANGE 4 5  I would appreciate a second pair of eyes on this. Is my method of 'locating the value' logic off? I thought the .loc/.iloc would be best but perhaps another way of indexing is best? I would be open to any solutions!

Answer: IMHO, the title makes your question an XY. I think you want to simply merge both objects : # sample_df["b"] = "" # no need for this line anymore out = sample_df.merge(pd.DataFrame(ex_list, columns=["a", "b"]), how="left")  Output : print(out) a b 0 1 CHANGE 1 2 NaN 2 3 NaN 3 4 CHANGE 4 5 NaN
Question: I have a binary N-by-N (N~=200) NumPy array populated with zeroes and ones. I would like to apply a Boolean mask and 'swap' the values that correspond to True in the mask, so for example if I had: arr = np.array([0,0,1,1], [0,0,1,1], [0,0,1,1], [0,0,1,1]) mask = np.array([True,False,True,False], [True,False,True,False], [True,False,True,False], [True,False,True,False],  I would like the resulting array to be: arr_new = np.array([1,0,0,1], [1,0,0,1], [1,0,0,1], [1,0,0,1])  I started this by initially creating a function swap_cell that swaps between the values, and then followed the approaches in this answer. def swap_cell(x): if x == 1.0: return 0.0 elif x == 0.0: return 1.0 arr_new = np.where(mask,swap_cell(arr),arr)  This code returns ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all(), which I understand is because of the 'if' statement in my swap_cell() function. I know there must be a much more pythonic way to accomplish this, but I thought this might work.

Answer: If you interpret the first (0,1) array as a boolean array, then you are doing the exclusive or operation of each entry in the first array with each entry of the second array. Therefore, you can do this with the numpy function np.logical_xor() as follows: arr = np.array([[0,0,1,1], [0,0,1,1], [0,0,1,1], [0,0,1,1]]) mask = np.array([[True,False,True,False], [True,False,True,False], [True,False,True,False], [True,False,True,False]]) np.array(np.logical_xor(arr, mask), dtype=int)  which returns array([[1, 0, 0, 1], [1, 0, 0, 1], [1, 0, 0, 1], [1, 0, 0, 1]])
Question: I am looking for a way to create a type alias for a Union that is typeable. In my opinion, there is no suitable default type in the typing module.  Typing.Iterable is an iterable, which could be a dict — but it shouldn't be. typing.Sequence is a sequence, which supports access by integer indices, but this doesn't include sets.   Imagine I have these variables (which could of course also be parameters of a method): x : list[str] | tuple[str] | set[str] | frozenset[str] y : list[str] | tuple[str] | set[str] | frozenset[str] z : list[int] | tuple[int] | set[int] | frozenset[int]  This is basically always the same. Therefore I would like to create a type alias for it. Something like this: my_type = list | tuple | set | frozenset  so that I can annotate them, like: x: my_type[str] = ["foo", "bar"] y: my_type[str] = ["baz", "bar"] z: my_type[int] = [1, 7, 42]  But this gives me this error: TypeError: There are no type variables left in list | tuple | set | frozenset  When I define my type with Union (my_type = typing.Union[list, tuple, set, frozenset]) instead of the | shorthand, I get: Traceback (most recent call last): File "/opt/pycharm-professional/plugins/python/helpers/pydev/pydevconsole.py", line 364, in runcode coro = func() File "<input>", line 4, in <module> File "/usr/lib/python3.10/typing.py", line 312, in inner return func(*args, **kwds) File "/usr/lib/python3.10/typing.py", line 1058, in __getitem__ _check_generic(self, params, len(self.__parameters__)) File "/usr/lib/python3.10/typing.py", line 228, in _check_generic raise TypeError(f"{cls} is not a generic class") TypeError: typing.Union[list, tuple, set, frozenset] is not a generic class  How can I create my type alias to make this work? It should work with python>=3.10.

Answer: If I understand your question correctly you need a type alias which is generic over some type _T. If that's the case, for python>=3.10 you can make use of TypeAlias & TypeVar. from typing import TypeAlias, TypeVar _T = TypeVar("_T") ListOrTupleOrSetOrFrozenSet: TypeAlias = list[_T] | tuple[_T] | set[_T] | frozenset[_T] x: ListOrTupleOrSetOrFrozenSet[str] = ["1"]  And in python 3.12 you could also use the builtin type parameter syntax which also support generic. type ListOrTupleOrSetOrFrozenSet[T] = list[T] | tuple[T] | set[T] | frozenset[T] x: ListOrTupleOrSetOrFrozenSet[str] = ["1"]  See also:  How do you alias a type in Python?
Question: I would like to pass an array which contains a list of numba compiled functions as a parameter into an njit method. In my attempt to do so, I encountered the following error: non-precise type array(pyobject, 1d, C). I'm able to pass a numba compiled function as parameter into an njit method, but i'm unable to do so if the numba compiled functions are stored in an array. Is there a way to pass an array of functions as parameter into njit function? or something along the lines of type casting the array so numba knows the content of the array np.array([function1, function2], dtype=function)? Here's a sample of my code import numpy as np from numba import njit @njit() def function1(x, y): return x > y @njit() def function2(x, y): return x < y @njit() def main(inputArray): print(inputArray[0](1,2)) print(inputArray[1](1,2)) functionArray = np.array([function1, function2]) main(functionArray)  Error message numba.core.errors.TypingError: Failed in nopython mode pipeline (step: nopython frontend) non-precise type array(pyobject, 1d, C) 

Answer: You can try to pass variable number of arguments to function (but still, you get a warning message): import numba T = numba.float64( numba.float64, numba.float64, ) @numba.njit(T) def function1(x, y): return x > y @numba.njit(T) def function2(x, y): return x < y @numba.njit def main(*inputArray): print(inputArray[0](1.0, 2.0)) print(inputArray[1](1.0, 2.0)) l = [function1, function2] main(*l)  Prints: 0.0 1.0   EDIT: Using numba.typed.List - this compiles without warning. import numba T = numba.float64( numba.float64, numba.float64, ) @numba.njit(T) def function1(x, y): return x > y @numba.njit(T) def function2(x, y): return x < y @numba.njit def main(inputArray): print(inputArray[0](1.0, 2.0)) print(inputArray[1](1.0, 2.0)) function_sig = numba.types.FunctionType(T) l = numba.typed.List.empty_list(function_sig, 2) l.append(function1) l.append(function2) main(l)  Prints: 0.0 1.0
Question: I am trying to compare columns from 2 dataframes and return the difference, ignoring case. Here is what I have so far: import pandas as pd if __name__ == "__main__": data1={'Name':['Karan','Rohit','Sahil','Aryan']} data2={'Name':['karan','Rohit','Sahil']} df1=pd.DataFrame(data1) df2=pd.DataFrame(data2) print(list(set(df1['Name']).difference(df2['Name'])))  This code prints ['Karan', 'Aryan']. How do I modify this to ignore case so that karan and Karan are recognized as a match and only Aryan is returned? I don't want to use the following because it returns aryan and I want to maintain the capitalization of the row. In my real case, they are not first names so it isn't as easy as making the first letter capitalized again after taking the difference.  print(list(set(df1['Name'].str.lower()).difference(df2['Name'].str.lower()))) 

Answer: To perform a case insensitive comparison, use str.casefold: print(list(set(df1['Name'].str.casefold()).difference(df2['Name'].str.casefold())))  If you want to keep the original case use boolean indexing with isin: df1.loc[~df1['Name'].str.casefold().isin(df2['Name'].str.casefold()), 'Name'].unique()  Output: array(['Aryan'], dtype=object)
Question: I have the following code which loads my pdf file generates embeddings and stores them in a vector db. I can then use it to preform searches on it. The issue is that every time i run it the embeddings are regrated and stored in the db along with the ones already created. Im trying to figurer out How to load an existing vector db into Langchain. rather then recreating them every time the app runs.  load it def load_embeddings(store, file): # delete the dir # shutil.rmtree(store) # I have to delete it or it just loads double data loader = PyPDFLoader(file) text_splitter = CharacterTextSplitter( separator="\n", chunk_size=1000, chunk_overlap=200, length_function=len, is_separator_regex=False, ) pages = loader.load_and_split(text_splitter) return DocArrayHnswSearch.from_documents( pages, GooglePalmEmbeddings(), work_dir=store + "/", n_dim=768 )  use it db = load_embeddings("linda_store", "linda.pdf") embeddings = GooglePalmEmbeddings() query = "Have I worked with Oauth?" embedding_vector = embeddings.embed_query(query) docs = db.similarity_search_by_vector(embedding_vector) for i in range(len(docs)): print(i, docs[i])  issue This works fine but if I run it again it just loads the file again into the vector db. I want it to just use the db after I have created it and not create it again. I cant seem to find a method for loading it I tried db = DocArrayHnswSearch.load("hnswlib_store/", embeddings)  But thats a no go.

Answer: Your load_embeddings function is recreating the database every time you call it. Here's why: 1. You're loading from PyPDFLoader every time ... # We don't need this when loading from store loader = PyPDFLoader(file) ...  2. from_documents(documents, embedding, **kwargs) ... # We don't need to pass pages when loading from store return DocArrayHnswSearch.from_documents( pages, GooglePalmEmbeddings(), work_dir=store + "/", n_dim=768 ) ...  Instead, you can try this: def query_vector_store(query): embeddings = OpenAIEmbeddings(openai_api_key=open_ai_key) vector_store = DocArrayHnswSearch.from_params(embeddings, "store/", 1536) embedding_vector = embeddings.embed_query(query) return vector_store.similarity_search_by_vector(embedding_vector)  I am using OpenAIEmbeddings() here but the same code should apply to GooglePalmEmbeddings() just make sure you update the value of the dimension. 1. DocArrayHnswSearch.from_params We're using DocArrayHnswSearch.from_params instead to load embeddings from the store (see here). This method does not expect the documents. 2. We're using our vector_store to perform similarity search As you can see from the query_vector_store(query: str) function above, we're not re-loading the documents from the PDF loader every time. Instead, we're just passing in our embeddings, work directory, and dimensions. 3. Usage You can use the method as such: query_vector_store('YOUR_QUERY'). Based on your for loop here: for i in range(len(docs)): print(i, docs[i])  You'll see the documents sorted by most similar. I hope this helps!
Question: the dataclasses module lets users make a dict from a dataclass reall conveniently, like this: from dataclasses import dataclass, asdict @dataclass class MyDataClass: ''' description of the dataclass ''' a: int b: int # create instance c = MyDataClass(100, 200) print(c) # turn into a dict d = asdict(c) print(d)  But i am trying to do the reverse process: dict -> dataclass. The best that i can do is unpack a dict back into the predefined dataclass. # is there a way to convert this dict to a dataclass ? my_dict = {'a': 100, 'b': 200} e = MyDataClass(**my_dict) print(e)  How can i achieve this without having to pre-define the dataclass (if it is possible) ?

Answer: You can use make_dataclass: from dataclasses import make_dataclass my_dict = {"a": 100, "b": 200} make_dataclass( "MyDynamicallyCreatedDataclass", ((k, type(v)) for k, v in my_dict.items()) )(**my_dict)
Question: I have a list of tuple and I'd like to convert the list into a string of tuple pair separted by comman. Not sure how to do it? For example if I have list of this a=[(830.0, 930.0), (940.0, 1040.0)]  I'd like to convert it to a string like this b="(830.0, 930.0), (940.0, 1040.0)"  a=[(830.0, 930.0), (940.0, 1040.0), (1050.0, 1150.0), (1160.0, 1260.0), (1270.0, 1370.0), (1380.0, 1480.0), (1490.0, 1590.0)] b=','.join(a) b ----> 2 b=','.join(a) 3 b TypeError: sequence item 0: expected str instance, tuple found 

Answer: Use f-strings or formatted string literals and str.strip: a = [(830.0, 930.0), (940.0, 1040.0)] b = f"{a}".strip("[]") print(b) # (830.0, 930.0), (940.0, 1040.0)
Question: Why is this priority queue failing to heapify? Where (150, 200, 200) are the priority values assigned to the dictionaries import heapq priority_q = [ (150, {'intel-labels': {'timestamp': 150}}), (200, {'intel-labels': {'timestamp': 200}}), (200, {'intel-labels': {'timestamp': 200, 'xx': 'xx'}}) ] heapq.heapify(priority_q) print( heapq.nlargest(2, priority_q))  The exception: Traceback (most recent call last): File "<stdin>", line 1, in <module> TypeError: '<' not supported between instances of 'dict' and 'dict'  The below, however, works.. priority_q = [ (150, {'intel-labels': {'timestamp': 150}}), (200, {'intel-labels': {'timestamp': 200}}), (201, {'intel-labels': {'timestamp': 200, 'xx': 'xx'}}) ] heapq.heapify(priority_q)  Why is this?

Answer:  Why is this?  heapq heapifies the heap according to comparisons. Tuples are compared lexicographically. If your first values in all tuples are distinct, you will never compare the second values. This is the case for your second example. If you have a duplicate value (200 in your first example), the second elements will be compared. These are dicts (which can't be compared), so this raises an error. As for a proper fix: The heapq docs got you covered. Their suggestion is to use triples such that the second value is an autoincremented tie breaker, or to use a data class that only compares the priority field. Note that these approaches differ slightly: With a tie breaker, there won't be any ties; nlargest will always give you just a single element - the one which was inserted first. If you want it to return all tying elements, you shouldn't use this approach. Applying the second approach to your example, the following works: import heapq from dataclasses import dataclass, field from typing import Any @dataclass(order=True) class PrioritizedItem: priority: int item: Any=field(compare=False) priority_q = [ PrioritizedItem(150, {'intel-labels': {'timestamp': 150}}), PrioritizedItem(200, {'intel-labels': {'timestamp': 200}}), PrioritizedItem(200, {'intel-labels': {'timestamp': 200, 'xx': 'xx'}}) ] heapq.heapify(priority_q) print(heapq.nlargest(2, priority_q))  prints [PrioritizedItem(priority=200, item={'intel-labels': {'timestamp': 200}}), PrioritizedItem(priority=200, item={'intel-labels': {'timestamp': 200, 'xx': 'xx'}})]