eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: Is there a way to find the position of the words with pos-tag 'NN' and 'VB' in a sentence in Python? example of a sentences in a csv file: "Man walks into a bar." "Cop shoots his gun." "Kid drives into a ditch"

Answer: You can find positions for certein PoS tags on a text using some of the existing NLP frameworks such us Spacy or NLTK. Once you process the text you can iterate for each token and check if the pos tag is what you are looking for, then get the start/end position of that token in your text. Spacy Using spacy, the code to implement what you want would be something like this: import spacy nlp = spacy.load("en_core_web_lg") doc = nlp("Man walks into a bar.") # Your text here words = [] for token in doc: if token.pos_ == "NOUN" or token.pos_ == "VERB": start = token.idx # Start position of token end = token.idx + len(token) # End position = start + len(token) words.append((token.text, start, end, token.pos_)) print(words)  In short, I build a new document from the string, iterate over all the tokens and keep only those whose post tag is VERB or NOUN. Finally I add the token info to a list for further processing. I strongly recommend that you read the following spacy tutorial for more information. NLTK Using NLTK I think is pretty straightforward too, using NLTK tokenizer and pos tagger. The rest is almost analogous to how we do it using spacy. What I'm not sure about is the most correct way to get the start-end positions of each token. Note that for this I am using a tokenization helper created by WhitespaceTokenizer().tokenize() method, which returns a list of tuples with the start and end positions of each token. Maybe there is a simpler and NLTK-like way of doing it. import nltk from nltk.tokenize import WhitespaceTokenizer text = "Man walks into a bar." # Your text here tokens_positions = list(WhitespaceTokenizer().span_tokenize(text)) # Tokenize to spans to get start/end positions: [(0, 3), (4, 9), ... ] tokens = WhitespaceTokenizer().tokenize(text) # Tokenize on a string lists: ["man", "walks", "into", ... ] tokens = nltk.pos_tag(tokens) # Run Part-of-Speech tager # Iterate on each token words = [] for i in range(len(tokens)): text, tag = tokens[i] # Get tag start, end = tokens_positions[i] # Get token start/end if tag == "NN" or tag == "VBZ": words.append((start, end, tag)) print(words)  I hope this works for you!
Question: Suppose I create two instances of class object. Are these two instances guaranteed to be not equal to each other? In other words, is object() == object() guaranteed to be False, or is it implementation-dependent? I understand that object() is object() is guaranteed to be False, but here I am asking about object() == object().

Answer: Yes it is guaranteed that object() == object() is False because it is documented that "by default, object implements __eq__() by using is".
Question: I have read on the documentation of Django about gettext_lazy, but there is no clear definition for me about this exact function, and I want to remove it. I found this implementation: from django.utils.translation import gettext_lazy as _  and it is used on a django model field in this way: email = models.EmailField(_('email address'), unique=True)  What is it for? What happens if I remove it?

Answer: Its used for translation for creating translation files like this: # app/locale/cs/LC_MESSAGES/django.po #: templates/app/index.html:3 msgid "email address" msgstr "emailová adresa"  Then it can be rendered in template as translated text. Nothing will happen if you remove it and don't want to use multilingualism.
Question: I have a dataframe that has many rows per combination of the 'PROGRAM', 'VERSION', and 'RELEASE_DATE' columns. I want to get a dataframe with all of the combinations of just those three columns. Would this be a job for groupby or distinct?

Answer: Since you are not aggregating anything, use unique df.select('PROGRAM','VERSION','RELEASE_DATE').unique()
Question: I am trying to convert images in folder named Final to one pdf This is my try to achieve the task import os import img2pdf with open("Stickers.pdf","wb") as f: #print(os.listdir("./Final")) f.write(img2pdf.convert([x for x in os.listdir("./Final") if x.endswith(".png")]))  But I got an error message like that (any idea how to fix that)  File "C:\Users\Future\Desktop\demo.py", line 5, in <module> f.write(img2pdf.convert([x for x in os.listdir("./Final") if x.endswith(".png")])) File "C:\Users\Future\AppData\Local\Programs\Python\Python39\lib\site-packages\img2pdf.py", line 2263, in convert ) in read_images( File "C:\Users\Future\AppData\Local\Programs\Python\Python39\lib\site-packages\img2pdf.py", line 1444, in read_images im = BytesIO(rawdata) TypeError: a bytes-like object is required, not 'str' 

Answer: The error message is cryptic largely because img2pdf bends over backwards to be so versatile about what you can pass to the convert function: a file path (as a string or path object), an open file object, raw image data, multiples of any of those, or a list of any of those. If given a string, it first treats it as a file path and tries to open it; if that fails, it then assumes the input must be raw image data (as bytes) and attempts to process it that way, in which case you get the TypeError that you see. The reason it can't open the files correctly is that os.listdir returns only the file names themselves, not their full path, and those image files aren't in your current working directory. That is, you're trying to open example.png, not ./Final/example.png. You can fix this by manually adding the folder name to each file name: import os import img2pdf with open("Stickers.pdf", "wb") as f: f.write(img2pdf.convert([f"./Final/{x}" for x in os.listdir("./Final") if x.endswith(".png")]))  You could also use pathlib. It often makes dealing with paths easier (and more easily platform-independent) than messing about with os. Path objects are smarter than plain strings, and automatically know about their enclosing folder in this case. All of Python's builtins (like open) will accept a Path object anywhere they already accept a string representing a path, and by this point practically all third-party libraries do as well. This includes img2pdf, as of 0.5.0. Here's what a very literal translation would look like, except using pathlib: from pathlib import Path import img2pdf with open("Stickers.pdf", "wb") as f: f.write(img2pdf.convert([path for path in Path("./Final").glob("*.png")]))  But now that the glob method's doing the filtering, the list comprehension isn't actually doing anything. We can't actually pass it the result of glob, as that's a generator rather than a list, but we can unpack it to pass the individual results as multiple arguments. from pathlib import Path import img2pdf with open("Stickers.pdf", "wb") as f: f.write(img2pdf.convert(*Path("./Final").glob("*.png")))
Question: I'd like to parse a Polygon with a list of Points out of the following data: {"points": ["0|0", "1|0", "1|1"]}  I naively thought I could do something like this: from pydantic import BaseModel, validator class Point(BaseModel): x: int y: int @validator("x", "y", pre=True) def get_coords(cls, value, values): x, y = value.split("|") values["x"] = x values["y"] = y class Polygon(BaseModel): points: list[Point]  But when I try and parse my "JSON" string I get an error complaining that value is not a valid dict: >>> Polygon.parse_obj({"points": ["0|0", "1|0", "1|1"]}) Traceback (most recent call last): File "<stdin>", line 1, in <module> File "pydantic/main.py", line 511, in pydantic.main.BaseModel.parse_obj File "pydantic/main.py", line 331, in pydantic.main.BaseModel.__init__ pydantic.error_wrappers.ValidationError: 3 validation errors for Polygon points -> 0 value is not a valid dict (type=type_error.dict) points -> 1 value is not a valid dict (type=type_error.dict) points -> 2 value is not a valid dict (type=type_error.dict)  How can I parse interesting objects out of this dull list of strings?

Answer: In Pydantic v2.0 it is possible to directly create custom conversions from arbitrary data to a BaseModel. This allows to define the conversion once for the specific BaseModel to automatically make containing classes support the conversion. The trick is to use a @model_validator(mode="before") to parse input before creating the model: from pydantic import BaseModel, model_validator class Point(BaseModel): x: int y: int @model_validator(mode="before") @classmethod def from_literal(cls, data: Any) -> Any: """Automatically parse 'x|y' literals""" if isinstance(data, str): # optionally validate literal assert data.count("|") == 1, "literal requires one | separator" x, y = data.split("|") # field type conversion can be handled by pydantic return dict(x=x, y=y) return data  Notably, we only have to parse the input into fields that Pydantic can convert to the final field type. This also allows to combine multiple models that define their own literal syntax. In addition, we can optionally validate the literal using the usual Pydantic means of asserting assumptions. Also, we should pass through other data to still support instances and data dictionaries.  This definition is sufficient to parse even nested literals automatically, while still accepting explicit instances and data: # these create the same objects print(Polygon.model_validate({"points": ["0|0", "1|0", "1|1"]})) print(Polygon.model_validate({"points": ["0|0", Point(x=1, y=0), {"x": 1, "y": 1}]})) # validation rejects the invalid literal print(Polygon.model_validate({"points": ["0|0", "1|0", "1|1|1"]}))  model_validate replaces parse_obj in Pydantic v2.0. However, this still works with parse_obj.
Question: I'd like to fix the random seed from BERTopic library to get reproducible results. Looking at the code of BERTopic I see it uses numpy. Will using np.random.seed(123) be enough? or do I also need to other libraries as random or pytorch as in this question.

Answer: You can fix the random_state variable using UMAP, but you have to also send the other default parameters to the UMAP constructor or the model will break. What this looks like in practice is: umap = UMAP(n_neighbors=15, n_components=5, min_dist=0.0, metric='cosine', low_memory=False, random_state=1337) model = BERTopic(language="multilingual", umap_model=umap) topics, probs = model.fit_transform(content)  By default, umap_model is set to None in the BERTopic constructor. Internally if that is not provided, it sets one up with default params here in the code. Note that low_memory is a param in both constructors, and if the BERTopic constructor isn't called with that in it, it internally sets it to False.
Question: I tried to list all files in a bucket. Here is my code import boto3 s3 = boto3.resource('s3') my_bucket = s3.Bucket('my_project') for my_bucket_object in my_bucket.objects.all(): print(my_bucket_object.key)  it works. I get all files' names. However, when I tried to do the same thing on a folder, the code raise an error import boto3 s3 = boto3.resource('s3') my_bucket = s3.Bucket('my_project/data/') # add the folder name for my_bucket_object in my_bucket.objects.all(): print(my_bucket_object.key)  Here is the error: botocore.exceptions.ParamValidationError: Parameter validation failed: Invalid bucket name "carlos-cryptocurrency-research-project/data/": Bucket name must match the regex "^[a-zA-Z0-9.\-_]{1,255}$" or be an ARN matching the regex "^arn:(aws).*:(s3|s3-object-lambda):[a-z\-0-9]*:[0-9]{12}:accesspoint[/:][a-zA-Z0-9\-.]{1,63}$|^arn:(aws).*:s3-outposts:[a-z\-0-9]+:[0-9]{12}:outpost[/:][a-zA-Z0-9\-]{1,63}[/:]accesspoint[/:][a-zA-Z0-9\-]{1,63}$"  I'm sure the folder name is correct and I tried replacing it with Amazon Resource Name (ARN) and S3 URI, but still get the error.

Answer: You can't indicate a prefix/folder in the Bucket constructor. Instead use the client-level API and call list_objects_v2 something like this: import boto3 client = boto3.client('s3') response = client.list_objects_v2( Bucket='my_bucket', Prefix='data/') for content in response.get('Contents', []): print(content['Key'])  Note that this will yield at most 1000 S3 objects. You can use a paginator if needed, or consider using the higher-level Bucket resource and its objects collection which handles pagination for you, per another answer to this question.
Question: I have a proxy class with an underlying object. I wish to pass the proxy object to a function that expects and underlying object type. How do I make the proxy typing match the underlying object? class Proxy: def __init__(self, obj): self.obj = obj def __getattribute__(self, name): return getattr(self.obj, name) def __setattr__(self, name, value): setattr(self.obj, name, value) def foo(bar: MyClass): ... foo(Proxy(MyClass())) # Warning: expected 'MyClass', got 'Proxy' instead 

Answer: Credit to @SUTerliakov for actually providing the essence of this answer in the comment on the question. The code should look something along the lines of the following: from typing import TYPE_CHECKING from my_module import MyClass if TYPE_CHECKING: base = MyClass else: base = object class Proxy(base): def __init__(self, obj): self.obj = obj def __getattribute__(self, name): return getattr(self.obj, name) def __setattr__(self, name, value): setattr(self.obj, name, value) def foo(bar: MyClass): ... foo(Proxy(MyClass())) # Works!  Note that your class may not always be available - but that's no real problem. The trick works perfectly fine with TypeVars. # proxy.py from typing import TYPE_CHECKING, TypeVar Proxied = TypeVar('Proxied') if TYPE_CHECKING: base = Proxied else: base = object class Proxy(base): def __init__(self, obj: Proxied): self.obj = obj def __getattribute__(self, name): return getattr(self.obj, name) def __setattr__(self, name, value): setattr(self.obj, name, value)  And in the file where you use it: from proxy import Proxy def foo(bar: MyClass): ... foo(Proxy(MyClass())) # Works!
Question: I am trying to make upload file input from swagger (with drf-yasg), but when I use MultiPartParser class it gives me the below error: drf_yasg.errors.SwaggerGenerationError: FileField is supported only in a formData Parameter or response Schema  My view: class AddExperience(generics.CreateAPIView): parser_classes = [MultiPartParser] permission_classes = [IsAuthenticated] serializer_class = DoctorExperienceSerializer  My serializer: class DoctorExperienceSerializer(serializers.Serializer): diploma = serializers.ListField( child=serializers.FileField(allow_empty_file=False) ) education = serializers.CharField(max_length=1000) work_experience = serializers.CharField(max_length=1000)  I also tried FormParser but it still gives me the same error. Also: FileUploadParser parser but it works like JsonParser:

Answer: The OpenAPISchema (OAS) 2 doesn't support the multiple file upload (see issue #254); but OAS 3 supports it (you can use this YML spec on a live swagger editer (see this result)). Comes to the real issue, there is a section in the drf-yasg's doc,  If you are looking to add Swagger/OpenAPI support to a new project you might want to take a look at drf-spectacular, which is an actively maintained new library that shares most of the goals of this project, while working with OpenAPI 3.0 schemas. OpenAPI 3.0 provides a lot more flexibility than 2.0 in the types of API that can be described. drf-yasg is unlikely to soon, if ever, get support for OpenAPI 3.0.  That means the package drf-yasg doesn't have support for OAS3 and thus, it won't support the "multiple file upload" feature. You can consider migrating from drf-yasg to drf-spectacular. But, also note that, drf-spectacular is also dealing the FileUpload in a different way.
Question: I’ve a lot of DataFrames with 2 columns, like this:      Fecha unidades     0 2020-01-01 2.0   84048 2020-09-01 4.0   149445 2020-10-01 11.0   532541 2020-11-01 4.0   660659 2020-12-01 2.0   1515682 2021-03-01 9.0   1563644 2021-04-01 2.0   1759823 2021-05-01 1.0   2226586 2021-07-01 1.0     As it can be seen, there are some months that are missing. Missing data depends on the DataFrame, I can have 2 months, 10, 100% complete, only one...I need to complete column "Fecha" with missing months (from 2020-01-01 to 2021-12-01) and when date is added into "Fecha", add "0" value to "unidades" column. Each element in Fecha Column is a class 'pandas._libs.tslibs.timestamps.Timestamp How could I fill the missing dates for each DataFrame??

Answer: You could create a date range and use "Fecha" column to set_index + reindex to add missing months. Then fillna + reset_index fetches the desired outcome: df['Fecha'] = pd.to_datetime(df['Fecha']) df = (df.set_index('Fecha') .reindex(pd.date_range('2020-01-01', '2021-12-01', freq='MS')) .rename_axis(['Fecha']) .fillna(0) .reset_index())  Output:  Fecha unidades 0 2020-01-01 2.0 1 2020-02-01 0.0 2 2020-03-01 0.0 3 2020-04-01 0.0 4 2020-05-01 0.0 5 2020-06-01 0.0 6 2020-07-01 0.0 7 2020-08-01 0.0 8 2020-09-01 4.0 9 2020-10-01 11.0 10 2020-11-01 4.0 11 2020-12-01 2.0 12 2021-01-01 0.0 13 2021-02-01 0.0 14 2021-03-01 9.0 15 2021-04-01 2.0 16 2021-05-01 1.0 17 2021-06-01 0.0 18 2021-07-01 1.0 19 2021-08-01 0.0 20 2021-09-01 0.0 21 2021-10-01 0.0 22 2021-11-01 0.0 23 2021-12-01 0.0
Question: I am writing a FastAPI app in python and I would like to use the openapi docs which are automatically generated. In particular, I would like to specify examples for the response value. I know how to do it when the response_model is a class that inherits from pydantic's BaseModel, but I am having trouble when it is a list of such classes. Here's a minimal example: from fastapi import FastAPI from typing import List from pydantic import BaseModel, Field class Person(BaseModel): name: str = Field( ..., title="Name", description="The name of the person", example="Alice" ) age: int = Field( ..., title="Age", description="The age of the person", example=83 ) class Config: schema_extra = { 'examples': [ { "name": "Alice", "age": 83 }, { "name": "Bob", "age": 77 } ] } app = FastAPI() @app.get('/person', response_model=Person) def person(): return { "name": "Alice", "age": 83 } @app.get('/people', response_model=List[Person]) def people(): return [ { "name": "Alice", "age": 83 }, { "name": "Bob", "age": 77 } ]  In the automatically generated openapi docs, the example value for a successful response for /person is { "name": "Alice", "age": 83 }  which is what I want. However, for /people it is [ { "name": "Alice", "age": 83 } ]  but I would prefer for it to be [ { "name": "Alice", "age": 83 }, { "name": "Bob", "age": 77 } ]  Is there any way to achieve that? Thank you in advance!

Answer: You can specify your example in responses parameter: @app.get('/people', response_model=List[Person], responses={ 200: { "description": "People successfully found", "content": { "application/json": { "example": [ { "name": "Alice", "age": 83 }, { "name": "Bob", "age": 77 } ] } } }, 404: {"description": "People not found"} }) def people(): return [ { "name": "Alice", "age": 83 }, { "name": "Bob", "age": 77 } ]  With this code, the example is specified for status code 200, but you can also configure example for errors (or you can remove 404 entry if you do not want it to appear in your openapi). For more information you can check FastAPI documentation: https://fastapi.tiangolo.com/advanced/additional-responses/
Question: Is there any way for me to apply a transformation on a ParamSpec? I can illustrate the problem with an example: from typing import Callable def as_upper(x: str): return x.upper() def eventually(f: Callable[P, None], *args: P.args, **kwargs: P.kwargs): def inner(): def transform(a): return a() if isinstance(a, Callable) else a targs = tuple(transform(a) for a in args) tkwargs = {k: transform(v) for k,v in kwargs.items()} return f(*targs, **tkwargs) return inner eventually(as_upper, lambda: "hello") # type checker complains here  This type checker (pyright in my case) will complain about this. The function eventually received a callable () -> str and not a str which was expected. My question is: is there some way for me to specify that it should expect () -> str and not the str itself? And in general, if a function expects a type T I can transform it to (say) () -> T? I'm basically asking if it's possible to transform the ParamSpec in some way so that a related function does not expect the same parameters, but "almost" the same parameters. I don't really expect this to be possible, but maybe some with more experience with type checking know a potential solution to this problem. :)

Answer: This decorator cannot be properly typed with currently available tools (Python 3.10). Two main problems here:  ParamSpec and Concatenate for now only allow us to modify a fixed number of parameters. We cannot concatenate keyword-only arguments (which makes transforming **kwargs: P.kwargs impossible)  However, under these constraints, we can achieve a less elegant solution if the positional parameters are known, taking advantage of currying: from typing import Callable, Concatenate, ParamSpec, TypeVar T = TypeVar("T") RetT = TypeVar("RetT") P = ParamSpec("P") def something(a: str, b: int, c: bool): return f"{a} {b} {c}" def eventually( f: Callable[Concatenate[T, P], RetT], a: Callable[[], T] ) -> Callable[P, RetT]: def inner(*args: P.args, **kwargs: P.kwargs) -> RetT: transformed = a() if callable(a) else a return f(transformed, *args, **kwargs) return inner something_eventually = eventually( eventually(eventually(something, lambda: "hello"), lambda: 2), lambda: False ) something_eventually() # hello 2 False  Notice that it is yet not possible to concatenate keyword parameters (See also). eventually can also be applied in a more functional way (but it cannot be properly typed): from functools import reduce # Though it unfortunately doesn't type check something_eventually = reduce( eventually, [lambda: "hello", lambda: 2, lambda: False], something, ) something_eventually() # hello 2 False  reduce expects the type of the value to have the same type, while we are changing the type of the function in each iteration. This makes it impossible to type it when we apply eventually arbitrary times in this manner. reduce is just an example. In a more general sense, I doubt if we can properly type something that involves repeatedly applying currying functions like eventually as proposed above, at least with the current Concatenate support.
Question: I need to listen to User Data Stream, whenever there's an Order Event - order execution, cancelation, and so on - I'd like to be able to listen to those events and create notifications. So I got my "listenKey" and I'm not sure if it was done the right way but I executed this code and it gave me something like listenKey. Code to get listenKey: def get_listen_key_by_REST(binance_api_key): url = 'https://api.binance.com/api/v1/userDataStream' response = requests.post(url, headers={'X-MBX-APIKEY': binance_api_key}) json = response.json() return json['listenKey'] print(get_listen_key_by_REST(API_KEY))  And the code to listen to User Data Stream - which doesn't work, I get no json response. socket = f"wss://fstream-auth.binance.com/ws/btcusdt@markPrice?listenKey=<listenKeyhere>" def on_message(ws, message): json_message = json.loads(message) print(json_message) def on_close(ws): print(f"Connection Closed") # restart() def on_error(ws, error): print(f"Error") print(error) ws = websocket.WebSocketApp(socket, on_message=on_message, on_close=on_close, on_error=on_error)  I have read the docs to no avail. I'd appreciate it if someone could point me in the right direction.

Answer: You can create a basic async user socket connection from the docs here along with other useful info for the Binance API. Here is a simple example: import asyncio from binance import AsyncClient, BinanceSocketManager async def main(): client = await AsyncClient.create(api_key, api_secret, tld='us') bm = BinanceSocketManager(client) # start any sockets here, i.e a trade socket ts = bm.user_socket() # then start receiving messages async with ts as tscm: while True: res = await tscm.recv() print(res) await client.close_connection() if __name__ == "__main__": loop = asyncio.get_event_loop() loop.run_until_complete(main())