eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I am trying to validate the latitude and longitude: from pydantic import BaseModel, Field from pydantic.dataclasses import dataclass @dataclass(frozen=True) class Location(BaseModel): longitude: float = Field(None, ge=-180, le=180) latitude: float = Field(None, ge=-90, le=90) Location(longitude=1.0, latitude=1.0)  When I run this locally I get the following error: Input should be a valid dictionary or instance of Location [type=model_type, input_value=ArgsKwargs((), {'longitud...: 1.0, 'latitude': 1.0}), input_type=ArgsKwargs] For further information visit https://errors.pydantic.dev/2.4/v/model_type  I have already tried to follow the Pydantic documentation which is given by the error but unfortunately is not clear and does not really give any information in regards of what to change and what to do differently. I also tried to create a Python dictionary to see if the error could be solved but it still gave the same error. Honestly, I have no idea of what to do next and how to solve this problem as I have tested it in several files (including in the same file to ensure it had nothing to do with any imports). Any ideas of why the error occurs? Update Without the (BaseModel) when running the an instance of the dataclass in the same file it works. Now, I am creating different unit tests to test if the validators work correctly. Please check the following code snippet: When creating a unit test and running a such test:  def test_when_location_is_not_in_range_then_print_exception(self): invalid_location = Location(latitude=200.0, longitude=300.0) with self.assertRaises(ValueError) as context: location = Location(latitude=invalid_location.latitude, longitude=invalid_location.longitude) self.assertEqual( str(context.exception), "Location is out of range", "Expected exception message: 'Location is out of range'" )  Before running the test, a warning appears: Unexpected Argument After running the test, a new error appears: longitude Input should be less than or equal to 180 [type=less_than_equal, input_value=300.0, input_type=float] For further information visit https://errors.pydantic.dev/2.4/v/less_than_equal latitude Input should be less than or equal to 90 [type=less_than_equal, input_value=200.0, input_type=float] For further information visit https://errors.pydantic.dev/2.4/v/less_than_equal  The error encountered before does not appear anymore.

Answer: I guess you're using dataclass from pydantic.dataclasses. In that case, don't inherit from BaseModel from pydantic import Field from pydantic.dataclasses import dataclass @dataclass(frozen=True) class Location: longitude: float = Field(None, ge=-180, le=180) latitude: float = Field(None, ge=-90, le=90) Location(longitude=1.0, latitude=1.0)  Read more at https://docs.pydantic.dev/latest/concepts/dataclasses/  Regarding the update to your question - the validation errors come from  invalid_location = Location(latitude=200.0, longitude=300.0)  You've limited the valid range of values for the latitude and longitude attributes, hence the errors when values provided do not adhere to those requirements.
Question: I have come across a different behaviour of search function in regex which made me think that there is an implicit \b anchor in the pattern. Is this the case? text = "bowl" print(re.search(r"b|bowl", text)) # first alteration in this pattern works print(re.search(r"o|bowl", text)) # but first alteration won't work here print(re.search(r"w|bowl", text)) # nor here print(re.search(r"l|bowl", text)) # nor here print(re.search(r"bo|bowl", text)) # first alteration in this pattern works print(re.search(r"bow|bowl", text)) # first alteration in this pattern works  OUTPUT <re.Match object; span=(0, 1), match='b'> <re.Match object; span=(0, 4), match='bowl'> <re.Match object; span=(0, 4), match='bowl'> <re.Match object; span=(0, 4), match='bowl'> <re.Match object; span=(0, 2), match='bo'> <re.Match object; span=(0, 3), match='bow'>  I have researched that if this was the case but I couldn't find any explanation.

Answer: I'm not a regex expert, so I'll use simple words to describe what happens internally. search works from left to right, and the | patterns too. Also search is different from match and moves forward to try to find the pattern across the string, not just at start. Take this: re.search(r"o|bowl", text)  So if o pattern is tested against, since matcher is on b character of the input string, it doesn't match, and the code tries the second pattern. If it failed, it would skip to next character (since all match possibilities are exhausted) and would match o, but since it matches, it doesn't happen: bowl characters are consumed. If you try: re.search("o|bar", text)  then o will be matched. Note that it's not specific to python. That's how a correct regex engine works. If you want the alternate behaviour you could write: re.search("o", text) or re.search("bar", text)
Question: If I have this minimal reproducible example import pandas as pd df = pd.DataFrame({"A":[12, 4, 5, None, 1], "B":[7, 2, 54, 3, None], "C":[20, 16, 11, 3, 8], "D":[14, 3, None, 2, 6]}) index_ = ['Row_1', 'Row_2', 'Row_3', 'Row_4', 'Row_5'] df.index = index_ print(df) # Option 1 result = df[['A', 'D']] print(result) # Option 2 result = df.loc[:, ['A', 'D']] print(result)  What is the effect on using loc or not. The results are quite similar. I ask this in preparation for a more complex question in which I have been instructed to use loc.

Answer: The difference is that df[['A', 'D']] is a weak reference to df (here on pandas 2.1.2). result1 = df[['A', 'D']] print(result1._is_copy) #<weakref at 0x7f34261b69d0; to 'DataFrame' at 0x7f34260e9590> result2 = df.loc[:, ['A', 'D']] print(result2._is_copy) # None  In both cases, this is not a view: print(result1._is_view, result2._is_view) # False False  This behavior has changed with the pandas versions. Is this important? It depends what you want to do. In most cases no. The first approach can however, in specific cases, trigger SettingWithCopyWarning: result1 = df[['A', 'D']] s1 = result1['A'] s1[:] = 1 # SettingWithCopyWarning: # A value is trying to be set on a copy of a slice from a DataFrame result2 = df.loc[:, ['A', 'D']] s2 = result2['A'] s2[:] = 1 # no Warning
Question: I have a dataframe that I'd like to break up into logical sub-dataframes. The most logical way to do this, given how the data is, is to select rows from the original dataframe where only one of the columns is not null (i.e. df.column.notnull() is True). Is there a shorthand for this or do I need to check each other column for every column that is not null?

Answer: Create a mask using .sum(axis=1) on the result of df.notnull() and checking if equal to 1: df[df.notnull().sum(axis=1).eq(1)]  With some sample data: import numpy as np import pandas as pd df = pd.DataFrame( {'A': [1, np.nan, np.nan, np.nan], 'B': [np.nan, 2, np.nan, np.nan], 'C': [np.nan, np.nan, 3, np.nan], 'D': [np.nan, np.nan, 4, np.nan]} ) df[df.notnull().sum(axis=1).eq(1)]  returns  A B C D 0 1.0 NaN NaN NaN 1 NaN 2.0 NaN NaN  from the original DataFrame  A B C D 0 1.0 NaN NaN NaN 1 NaN 2.0 NaN NaN 2 NaN NaN 3.0 4.0 3 NaN NaN NaN NaN
Question: I'm trying to build an app with streamlit that uses Qdrant python client. to run the qdrant, im just using: docker run -p 6333:6333 qdrant/qdrant  I have wrapped the client in something like this: class Vector_DB: def __init__(self) -> None: self.collection_name = "__TEST__" self.client = QdrantClient("localhost", port=6333,path = "/home/Desktop/qdrant/qdrant.db")  but i'm getting this error:  Storage folder /home/Desktop/qdrant/qdrant.db is already accessed by another instance of Qdrant client. If you require concurrent access, use Qdrant server instead.  I suspect that streamlit is creating multiple instances of this class, but, if i try to load the db from one snapshot, like:  class Vector_DB: def __init__(self) -> None: self.client = QdrantClient("localhost", port=6333) self.client.recover_snapshot(collection_name = "__TEST__",location = "http://localhost:6333/collections/__TEST__/snapshots/__TEST__-8742423504815750-2023-10-30-12-04-14.snapshot")  it works. Seems like i'm missing something important on how to configure it. What is the properly way of setting Qdrant, to store some embeddings, turn off the machine, and reload it?

Answer: You mention using the Qdrant server, to which you'd like to connect with the Python client. There are two problems in your above question, let me go over both of them: 1. Persist data in Qdrant server: A Qdrant server stores its data inside the Docker container. Docker containers are immutable however, which means that they don't hold data across restarts. To persist data you must specify a mount. Qdrant will then persist data on the mount instead of in the immutable container. You could configure a mount using the -v flag like this1: docker run -p 6333:6333 \ -v $(pwd)/qdrant_storage:/qdrant/storage:z \ qdrant/qdrant  Data is automatically persisted and reloaded when you stop or restart the Qdrant container. You don't have to take extra measures for this. 2. Qdrant server versus local mode: Qdrant supports two operating modes. The Qdrant server and local mode. You're using the Qdrant server through Docker. The Python client also supports local mode which is an in-memory implementation intended for testing. To use a Qdrant server you must specify its location (URL)2. You've already specified "localhost", perfect if hosting the Qdrant server on your local machine. To use local mode you can either specify ":memory:" or provide a path to persist data3. Right now you've specified parameters for both. Instead you must stick with one. You can update your client initialization to this: class Vector_DB: def __init__(self) -> None: self.collection_name = "__TEST__" self.client = QdrantClient("localhost", port=6333)
Question: Given the following code, I think output (A) makes sense since __str__ takes precedence over __repr__ however I am a bit confused about output (B) why does it favour __repr__ over __str__ and is there a way to make the class use the __str__ rather than the __repr__ of Foo without defining __str__ for Bar? @dataclass() class Foo(): def __repr__(self) -> str: return "Foo::__repr__" def __str__(self) -> str: return "Foo::__str__" @dataclass() class Bar(): foo: Foo f = Foo() b = Bar(f) print(f) # (A) Outputs: Foo::__str__ print(b) # (B) Outputs: Bar(foo=Foo::__repr__)  I stumbled upon this because I saw that my custom __str__ of Foo was not being used by Bar in fact even if I remove the __repr__ output (B) will still be Bar(foo=Foo()) instead of Bar(foo=Foo::__str__).

Answer: It is normal for the repr to be preferred to str when rendering a representation within a container of some sort, e.g.: >>> print(f) Foo::__str__ >>> print([f]) [Foo::__repr__]  It is necessary for repr to be unambiguous, for example to see the difference between numbers and strings here: >>> from dataclasses import dataclass >>> @dataclass ... class D: ... s: str ... n: int ... >>> D('1', 1) D(s='1', n=1)  I think you should not override Foo.__repr__ in that way, because you're straying from the "unwritten rule" that eval(repr(obj)) == obj.
Question: The result in the two following examples is different: EXAMPLE 1 a = 845 b = int("8"+"4"+"5") print(a == b) # True print(a is b) # False  EXAMPLE 2 a = 845 b = 840+5 print(a == b) # True print(a is b) # True  How can it be explained? Why in the first case the same integer is kept in two different memory slots in the pile? I expected the result to be the same (True) in both cases.

Answer: Another question provides good context on this issue, but this is a bit different to the cases mentioned there (but it comes down to the same thing in the end). Even though only integers between -5 and 256 are cached and return True for is checks, the same happens for hardcoded integers, since they are compiled as constants and are not allocated. Using compiler explorer makes this a bit easier to understand. First example can also be written as a = 845 b = int("845") print(a == b) # True print(a is b) # False  We can see in https://godbolt.org/z/zM3haedb9 that string concatenation is done in compile-time, but int is called in the end again, which creates a new integer and since it is not in the cached range, a new object is created, making is return False. For second example, addition is done in compile team, making it equivalent to a = 845 b = 845 print(a == b) # True print(a is b) # True  Which with the help of the other question makes complete sense. You can check out https://godbolt.org/z/xKacd5oh4 to see that after compilation, both a and b are created the same way.
Question: I am trying to make an app for electric vehicle drivers and i'm using a text file to store the data the way it works is i have the name of the electric vehicle and the the line under the name contains the miles it can get per 1%, i've got it so it can find the specific car but i can't find the range of the vehicle using that number. cars.txt MG MG4 EV Long Range 2.25 BMW iX1 xDrive30 2.3 Kia Niro EV 2.4 Tesla Model Y Long Range Dual Motor 2.7 BMW i4 eDrive40 3.2  code with open('cars.txt', 'r')as cars: check = input("Enter full name of car: ") car = cars.read() percentage = cars.readline() if check in car: print("Found") total = range print(percentage)  this is what i have but every time it finds the car it won't find the range after it.

Answer: You can do the following: target_car = "Kia Niro EV" with open("temp.txt") as f: for line in f: if line.rstrip() == target_car: range_ = float(next(f)) break else: range_ = "Not Found" print(f"range is: {range_}")  f is a consumable iterator. You iterate over it until you find your car, then the next item in that iterator is what you're looking for. Also note that you don't store the whole file in the memory in case you're dealing with a huge file. (In that case why wouldn't you use a proper database?)
Question: I have got data that looks like this data = [['01/01/2000', 'aaa', 101, 102], ['01/02/2000', 'aaa', 201, 202], ['01/01/2000', 'bbb', 301, 302], ['01/02/2000', 'bbb', 401, 402],] df = pd.DataFrame(data, columns=['date', 'id', 'val1', 'val2']) df date id val1 val2 01/01/2000 aaa 101 102 01/02/2000 aaa 201 202 01/01/2000 bbb 301 302 01/02/2000 bbb 401 402  I would like this data to be transformed to look like this - where it's grouped horizonally by the id column  aaa bbb date val1 val2 val1 val2 01/01/2000 101 102 301 302 01/02/2000 201 202 401 402  Closest i have gotten so far is: df.set_index(['date', 'id']).unstack(level=1), but this does not quite do it: val1 val2 id aaa bbb aaa bbb date 01/01/2000 101 301 102 302 01/02/2000 201 401 202 402 

Answer: Add DataFrame.swaplevel with DataFrame.sort_index: out = (df.set_index(['date', 'id']) .unstack(level=1) .swaplevel(0,1, axis=1) .sort_index(axis=1)) print (out ) id aaa bbb val1 val2 val1 val2 date 01/01/2000 101 102 301 302 01/02/2000 201 202 401 402  Or use DataFrame.melt with DataFrame.pivot and DataFrame.sort_index: out = (df.melt(['date', 'id']) .pivot(index='date', columns=['id', 'variable'], values='value') .sort_index(axis=1))
Question: I'm just studying python OOP, and truly confused when to use self and not. especially when I want to make a method that defaultly get object instance input and also wanna make it work as a normal method that can get the input of custom parameters, I get somewhat bothersome to type if x is None: x = self.x for all the parameters of the method. Example is as follows. from dataclasses import dataclass @dataclass class PlusCalculator(): x: int = 1 y: int = 2 result: int = None def plus(self, x=None, y=None): if x is None: x = self.x ## the bothersome part if y is None: y = self.y result = x + y self.result = result ## also confusing...is it good way? return result pc = PlusCalculator() print(pc.plus()) print(pc.plus(4,5))  Is there any good way to use instance variables as default value of function parameter??

Answer: A conditional expression is readable, fast, and intuitive- x = self.x if x is None else x  Re:  Is there any good way to use instance variables as default value of function parameter??  Regarding the setting of self.result- This should be avoided unless you need to access it as an instance variable later. As such you can simplify this to: @dataclass class PlusCalculator(): ... def plus(self, x=None, y=None): x = self.x if x is None else x y = self.y if y is None else y return x + y
Question: I have about a million files in my directory, and their number is likely to grow. For machine learning, I would like to randomly sample from those files without replacement. How can I do this very quickly? os.listdir(path) is too slow for me.

Answer:  I have about a million files in my directory ... os.listdir(path) is too slow for me.  This is the core of your problem, and it's solved by a technique I've generally heard referred to as bucketing your files, though a web search for this doesn't seem particularly helpful. Bucketing is generally used by programs that need to store a large number of files that don't have any particular structure - for example, all the media files (such as images) in a MediaWiki instance (the software that runs Wikipedia). Here's the Stack Overflow logo on Wikipedia: https://upload.wikimedia.org/wikipedia/commons/0/02/Stack_Overflow_logo.svg See that 0/02 in the URL? That's the bucket. All files in Wikipedia will be hashed by some algorithm - for example sha256, though it won't necessarily be this - and 02 will be the first two hex digits of that hash. (The 0 before the slash is just the first digit of 02; in this case it's used as a second level of bucketing.) If MediaWiki just stored every single file in one massive directory, it'd be very slow to access the files in that directory, because although OS folders can hold arbitrarily many files, they just aren't designed to hold more than a few thousand or so. By hashing the contents of the file, you get what looks like a random string of hex digits unique to that file, and if you then put all the files that start with the same first two hex digits (like 02 in a folder called 02, you get 256 folders (one for each possible value of the first two hex digits), and critically, each of those 256 folders contains a roughly equal number of files. When you're trying to look up particular files, like MediaWiki is, you obviously need to know the hash to get to the file, if you store it in this way. But in your case, you just want to load random files. So this will work just as well:  Hash all your files and bucket them (possibly with additional levels, e.g. you might want files like 12/34/filename.ext, so that you have 65,536 buckets). You can use things like hashlib or command-line tools like sha256sum to obtain file hashes. You don't need to rename the files, as long as you group them into directories based on the first few hex digits of their hashes. Now, each time you want a random file, choose a random bucket (and possibly random sub-buckets, if you're using additional levels), then choose a random file within that bucket.  Doing that will be a lot faster than using listdir on a directory with a million files and then choosing randomly among those.  Note: I'm just using MediaWiki as an example here because I'm familiar with a few of its internals; lots of software products do similar things.
Question: I create an array of bytes. array = bytearray(random.randint(1, 8192)) # Now, how can I randomize array's elements?  Now how can I randomize each elements of the array? Just like, // with Java var array = new byte[ThreadLocalRandom.current().nextInt(1, 8192)]; ThreadLocalRandom.current().nextBytes(array); 

Answer: def randomize(array): array[:] = random.randbytes(len(array))  Behaves like your answer's but is ~100 times faster. Time with an average-length array (4096 bytes):  23.0 ± 0.3 μs randomize_Kelly 2191.5 ± 11.6 μs randomize_Jin Python: 3.11.4 (main, Sep 9 2023, 15:09:21) [GCC 13.2.1 20230801]  Attempt This Online!
Question: I have a question on how to disable a nested context manager. I have an inside context manager: class cast: def __init__(self, enabled: bool = True, dtype) -> None: self.prev = False self.enabled = enabled self.dtype = dtype def __enter__(self) -> None: self.prev = is_cast_enabled() set_cast_enabled(self.enabled) set_cast_dtype(self.dtype) def __exit__(self, exc_type: Any, exc_value: Any, traceback: Any) -> None: set_cast_enabled(self.prev)  User can call the APIs to cast a region of code to a specific data type, with cast(dtype=numpy.float16): c = a + b  with this context manager, self.enabled=True and the cast is done. For example, here if both a and b are in float32, the output will be float16 since the cast is done. However, I wanted to a make another context manager, with no_cast, So with no_cast: with cast(dtype=numpy.float16): c = a + b  So that no matter user set the enabled to either True or False, the nested context manager won't work. Is that doable? I wanted to know how to disable a nested context manager.

Answer: The no_cast context manager can redefine cast so it doesn't do anything while in this context. class no_cast: def __enter__(self): global cast self.cast = cast cast = no_cast.do_nothing return self def __exit__(self, *args): global cast cast = self.cast return True def do_nothing(*args): pass
Question: I'm plotting a 3d bar plot for an array using matplotlib. I need to add an edgecolor to the bars. However, the edgecolor is coloring the 0.0 values data points in black. Is there a way to not color these data points? I'm trying to do edgecolors=none through a loop when values are 0.0. However this doesn't seem to help. Any help appreciated. The script I use to plot, from mpl_toolkits.mplot3d import Axes3D fig = plt.figure() ax = Axes3D(fig) lx= len(r[0]) ly= len(r[:,0]) xpos = np.arange(0,lx,1) ypos = np.arange(0,ly,1) xpos, ypos = np.meshgrid(xpos+0.25, ypos+0.25) xpos = xpos.flatten() ypos = ypos.flatten() zpos = np.zeros(lx*ly) cs = ['r', 'g', 'b', 'y', 'c'] * ly edgecolor_store = [] dx = 0.5 * np.ones_like(zpos) dy = 0 * np.ones_like(zpos) #dz = zpos dz = r.flatten() for value in dz: if value == 0.0: edgecolor_store.append('none') if value > 0.0: edgecolor_store.append('black') mask_dz = dz == 0 # SO:60111736, 3d case #print(dz) ax.bar3d(xpos,ypos,zpos, dx, dy,dz,color=cs,zsort='average',alpha=0.5,edgecolors=edgecolor_store)  Sample data I use, [ 0. , 0. , 0. , 0. , 0. ],[ 0. , 0. , 0. , 0. , 0. ],[ 0. , 0. , 0. , 0. , 0. ],[ 0. , 0. , 0. , 0. , 0. ],[ 0. , 0. , 0. , 0. , 0. ],[ 0. , 0. , 0. , 0. , 0. ],[ 0. , 0. , 20.7, 0. , 0. ],[ 0.6, 0. , 0.1, 0. , 0.2],[ 0. , 24.8, 0. , 46.7, 0. ],[ 0. , 0. , 99.7, 17.1, 99.3],[ 0. , 12.8, 98.6, 0. , 6.7],[ 0.2, 0. , 0. , 12.6, 0. ]  The output plot I get is, 

Answer: I feel that you've already tried the correct solution: not plot the unwanted bars. You need a mask. Btw, never ever compare floats with ==. mask = ~np.isclose(dz, 0.0)  Then, plot data filtered by this mask ax.bar3d(xpos[mask],ypos[mask],zpos[mask], dx[mask], dy[mask],dz[mask],color=cs[:mask.sum()],zsort='average',alpha=0.5,edgecolors='black')  Note that you no longer need edgecolor. And that my way of computing cs (starting from your too long cs, and truncating it to the exact needed size) is not optimal.  Also note that the 4 remaining black lines are not 0, but very small (0.1, 0.2, 0.2 and 0.6) values.