eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I'm unable to use python within the virtual environment. Python works fine outside of the virtual environment. I'm using Python 3.10.2 I keep on getting the error below when trying to run any python commands. 'C:\Users\User\AppData\Local\Programs\Python\Python310\python.exe'  It might be relevant to mention that I was unable to create the virtual environment through the 'python -m venv env' command. Error generated was Error: Command '['C:\\Users\\User\\Documents\\Python Projects\\PDFtoText\\env\\Scripts\\python.exe', '-Im', 'ensurepip', '--upgrade', '--default-pip']' returned non-zero exit status 101.  I had to add 'without-pip' to the end of the command to create the virtual environment. Weird thing is, I was able to use pip within the virtual environment without having to manually install it. The path to python is in the environmental variables. I tried reinstalling python but that did not help. Lastly, all these errors started occurring after I downloaded Visual Studio Community 2022.

Answer: Short answer, I bet you have a space in your Window's account name (say Your Account is where your account is saved so you have C:\Users\Your Account folder, and there is also a text file C:\Users\Your ("Your" being the first part of your user name). MSVS2022 (maybe earlier versions, too) is known to leave this log file which exposes a bug in Python venv's python launcher. Delete this text file, and your problem should be solved. See my question/answer for more details.
Question: I have fine-tuned my models with GPU but inferencing process is very slow, I think this is because inferencing uses CPU by default. Here is my inferencing code: txt = "This was nice place" model = transformers.BertForSequenceClassification.from_pretrained(model_path, num_labels=24) tokenizer = transformers.BertTokenizer.from_pretrained('TurkuNLP/bert-base-finnish-cased-v1') encoding = tokenizer.encode_plus(txt, add_special_tokens = True, truncation = True, padding = "max_length", return_attention_mask = True, return_tensors = "pt") output = model(**encoding) output = output.logits.softmax(dim=-1).detach().cpu().flatten().numpy().tolist()  Here is my second inferencing code, which is using pipeline (for different model): classifier = transformers.pipeline("sentiment-analysis", model="distilbert-base-uncased-finetuned-sst-2-english") result = classifier(txt)  How can I force transformers library to do faster inferencing on GPU? I have tried adding model.to(torch.device("cuda")) but that throws error: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu  I suppose the problem is related to the data not being sent to GPU. There is a similar issue here: pytorch summary fails with huggingface model II: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu How would I send data to GPU with and without pipeline? Any advise is highly appreciated.

Answer: You should transfer your input to CUDA as well before performing the inference: device = torch.device('cuda') # transfer model model.to(device) # define input and transfer to device encoding = tokenizer.encode_plus(txt, add_special_tokens=True, truncation=True, padding="max_length", return_attention_mask=True, return_tensors="pt") encoding = encoding.to(device) # inference output = model(**encoding)  Be aware nn.Module.to is in-place, while torch.Tensor.to is not (it does a copy!).
Question: So I lately came across an explanation for Python's interpreter and compiler (CPython specifically). Please correct me if I'm wrong. I just want to be sure I understand these specific concepts. So CPython gets both compiled (to bytecode) and then interpreted (in the PVM)? And what does the PVM do exactly? Does it read the bytecode line by line, and translate each one to binary instructions that can be executed on a specific computer? Does this mean that a computer based on an Intel processor needs a different PVM from an AMD-based computer?

Answer:  Yes, CPython is compiled to bytecode which is then executed by the virtual machine. The virtual machine executes instructions one-by-one. It's written in C (but you can write it in another language) and looks like a huge if/else statement like "if the current instruction is this, do this; if the instruction is this, do another thing", and so on. Instructions aren't translated to binary - that's why it's called an interpreter.  You can find the list of instructions here: https://docs.python.org/3.10/library/dis.html#python-bytecode-instructions The implementation of the VM is available here: https://github.com/python/cpython/blob/f71a69aa9209cf67cc1060051b147d6afa379bba/Python/ceval.c#L1718   Bytecode doesn't have a concept of "line": it's just a stream of bytes. The interpreter can read one byte at a time and use another if/else statement to decide what instruction it's looking at. For example: curr_byte = read_byte() if curr_byte == 0x00: # Parse instruction with no arguments curr_instruction = DO_THING_A; args = NULL; elif curr_byte == 0x01: another_byte = read_byte() if another_byte == 0x00: # Parse a two-byte instruction curr_instruction = DO_THING_B; args = NULL; else: # Parse a one-byte instruction # with one argument curr_instruction = DO_THING_C; args = another_byte >> 1; # or whatever elif curr_byte == ...: ... # go on and on and on   The entire point of bytecode is that it can be executed by another program (the interpreter, or virtual machine) on almost any hardware. For example, in order to get CPython running on new hardware, you'll need a C toolchain (compiler, linker, assembler etc) for this hardware and a bunch of functions that Python can call to do low-level stuff (allocate memory, output text, do networking etc). Once you have that, write C code that can execute the bytecode - and that's it.
Question: I am trying to write a df to an existing table with pandas.to_sql with this code: import sqlalchemy #CREATE CONNECTION constring = "mssql+pyodbc://UID:PASSWORD@SERVER/DATABASE?driver=SQL Server" dbEngine = sqlalchemy.create_engine(constring, fast_executemany=True, connect_args={'connect_timeout':10}, echo=False) #WRITE INTO TABLE df.to_sql(con=dbEngine, schema="dbo", name="target_table", if_exists="replace", index=False, chunksize=1000)  But it gives the following error: ProgrammingError: (pyodbc.ProgrammingError) ('42000', "[42000] [Microsoft][ODBC SQL Server Driver][SQL Server]CREATE TABLE permission denied in database 'database'. (262) (SQLExecDirectW)") [SQL: CREATE TABLE [target_table] ( [chLocalIndustryCode] VARCHAR(max) NULL, [vcLocalIndustryDesc] VARCHAR(max) NULL, [chLocalSectorCode] VARCHAR(max) NULL, [vcLocalSectorDesc] VARCHAR(max) NULL, [chLocalClusterCode] VARCHAR(max) NULL, [vcLocalClusterDesc] VARCHAR(max) NULL, [chLocalMegaClusterCode] VARCHAR(max) NULL, [vcLocalMegaClusterDesc] VARCHAR(max) NULL) ] (Background on this error at: https://sqlalche.me/e/14/f405)  I have checked the connection and works properly when reading the table , so I assume the problem is with pandas to_sql() function. I have also tried writing into the table with a cursor, but takes to long. I am using pandas 1.3.4 Is there any way of fixing this error or any alternative to pd.to_sql() function that I can use to increase writing speed?

Answer: You might need to add create permission to the SQL Server user. You can follow below steps from the link:  To add a Windows user that has the login “domainname \username” to the sysadmin fixed server role a. Log on to the computer using the credentials for the domainname\username account. b. Click the Start button, point to All Programs, click Microsoft SQL Server, right-click SQL Server Management Studio, and then click Run as administrator. ps: "Run As Administrator" option elevates the user permissions In the User Access Control dialog box, click Continue. c. In SQL Server Management Studio, connect to an instance of SQL Server. d. Click Security, right-click Logins, and then click New Login. e. In the Login name box, enter the user name. f. In the Select a page pane, click Server Roles, select the sysadmin check box, and then click OK.
Question: It is very surprising to me that I can instantiate an abstract class in python: from abc import ABC class Duck(ABC): def __init__(self, name): self.name = name if __name__=="__main__": d = Duck("Bob") print(d.name)  The above code compiles just fine and prints out the expected result. Doesn't this sort of defeat the purpose of having an ABC?

Answer: If you have no abstract method, you will able to instantiate the class. If you have at least one, you will not. Consider the following code: from abc import ABC, abstractmethod class Duck(ABC): def __init__(self, name): self.name = name @abstractmethod def implement_me(self): ... if __name__=="__main__": d = Duck("Bob") print(d.name)  which returns: TypeError: Can't instantiate abstract class Duck with abstract method implent_me  It is the combination of the ABC metaclass, and at least one abstractmethod that will lead to you not being able to instantiate a class. If you leave out one of the two, you will be able to do so.
Question: I would like to pass a JSON object to a FastAPI backend. Here is what I am doing in the frontend app: data = {'labels': labels, 'sequences': sequences} response = requests.post(api_url, data = data)  Here is how the backend API looks like in FastAPI: @app.post("/api/zero-shot/") async def Zero_Shot_Classification(request: Request): data = await request.json()  However, I am getting this error: json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0) 

Answer: You should use the json parameter instead (which would change the Content-Type header to application/json): payload = {'labels': labels, 'sequences': sequences} r = requests.post(url, json=payload)  not data which is used for sending form data with the Content-Type being application/x-www-form-urlencoded by default, or multipart/form-data if files are also included in the request—unless you serialized your JSON first and manually set the Content-Type header to application/json, as described in this answer: payload = {'labels': labels, 'sequences': sequences} r = requests.post(url, data=json.dumps(payload), headers={'Content-Type': 'application/json'})  Also, please have a look at the documentation on how to benefit from using Pydantic models when sending JSON request bodies, as well as this answer and this answer for more options and examples on how to define an endpoint expecting JSON data.
Question: I have the following FastAPI backend: from fastapi import FastAPI app = FastAPI class Demo(BaseModel): content: str = None @app.post("/demo") async def demoFunc(d:Demo): return d.content  The issue is that when I send a request to this API with extra data like: data = {"content":"some text here"}aaaa  or data = {"content":"some text here"aaaaaa} resp = requests.post(url, json=data)  it throws an error with status code 422 unprocessable entity error with Actual("some text here") and Extra("aaaaa") data in the return field in case of data = {"content":"some text here"}aaaa: { "detail": [ { "loc": [ "body", 47 ], "msg": "Extra data: line 4 column 2 (char 47)", "type": "value_error.jsondecode", "ctx": { "msg": "Extra data", "doc": "{\n \"content\": \"some text here\"}aaaaa", "pos": 47, "lineno": 4, "colno": 2 } } ] }  I tried to put the line app=FastAPI() in a try-catch block, however, it doesn't work. Is there any way I can handle this issue with own response instead of the above mentioned auto response? Something like this: {"error": {"message": "Invalid JSON body"}, "status": 0} 

Answer: You are passing an invalid JSON, and hence, the server correctly responds with the 422 Unprocessable Entity error. Your test client shouldn't be able to run at all, without throwing an invalid syntax error. So, I'm guessing you posted the request through the interactive autodocs provided by Swagger UI at /docs, and received the relevant 422 error. If what you actually want is to handle the error, in order to customize the error or something, you can override the request validation exception handler, as described in the documentation (have a look at this discussion, as well as this answer and this answer that demonstrates how to customize the RequestValidationError for specific routes only). Working Example: from fastapi import FastAPI, Body, Request, status from fastapi.encoders import jsonable_encoder from fastapi.exceptions import RequestValidationError from fastapi.responses import JSONResponse from pydantic import BaseModel app = FastAPI() class Demo(BaseModel): content: str = None @app.exception_handler(RequestValidationError) async def validation_exception_handler(request: Request, exc: RequestValidationError): return JSONResponse( status_code=status.HTTP_422_UNPROCESSABLE_ENTITY, content=jsonable_encoder({"detail": exc.errors(), # optionally include the errors "body": exc.body, "custom msg": {"Your error message"}}), ) @app.post("/demo") async def some_func(d: Demo): return d.content  Or, you could also return a PlainTextResponse with a custom message: from fastapi.responses import PlainTextResponse @app.exception_handler(RequestValidationError) async def validation_exception_handler(request, exc): return PlainTextResponse(str(exc), status_code=422)
Question: Is there an elegant way how to recode values in polars dataframe. For example 1->0, 2->0, 3->1...  in Pandas it is simple like that: df.replace([1,2,3,4,97,98,99],[0,0,1,1,2,2,2]) 

Answer: Edit 2024-07-09 Polars has dedicated replace and replace_strict expressions. df = pl.DataFrame({ "a": [1, 2, 3, 4, 5] }) mapper = { 1: 0, 2: 0, 3: 10, 4: 10 } df.select( pl.all().replace(mapper) )  shape: (5, 1) ┌─────┐ │ a │ │ --- │ │ i64 │ ╞═════╡ │ 0 │ │ 0 │ │ 10 │ │ 10 │ │ 5 │ └─────┘  Before Edit In polars you can build columnar if else statetements called if -> then -> otherwise expressions. So let's say we have this DataFrame. df = pl.DataFrame({ "a": [1, 2, 3, 4, 5] })  And we'd like to replace these with the following values: from_ = [1, 2] to_ = [99, 12]  We could write: df.with_columns( pl.when(pl.col("a") == from_[0]) .then(to_[0]) .when(pl.col("a") == from_[1]) .then(to_[1]) .otherwise(pl.col("a")).alias("a") )  shape: (5, 1) ┌─────┐ │ a │ │ --- │ │ i64 │ ╞═════╡ │ 99 │ │ 12 │ │ 3 │ │ 4 │ │ 5 │ └─────┘  Don't repeat yourself Now, this becomes very tedious to write really fast, so we could write a function that generates these expressions for use, we are programmers aren't we! So to replace with the values you have suggested, you could do: from_ = [1,2,3,4,97,98,99] to_ = [0,0,1,1,2,2,2] def replace(column, from_, to_): # initiate the expression with `pl.when` branch = pl.when(pl.col(column) == from_[0]).then(to_[0]) # for every value add a `when.then` for (from_value, to_value) in zip(from_, to_): branch = branch.when(pl.col(column) == from_value).then(to_value) # finish with an `otherwise` return branch.otherwise(pl.col(column)).alias(column) df.with_columns(replace("a", from_, to_))  Which outputs: shape: (5, 1) ┌─────┐ │ a │ │ --- │ │ i64 │ ╞═════╡ │ 0 │ │ 0 │ │ 1 │ │ 1 │ │ 5 │ └─────┘
Question: PEP 647 introduced type guards to perform complex type narrowing operations using functions. If I have a class where properties can have various types, is there a way that I can perform a similar type narrowing operation on the property of an object given as the function argument? class MyClass: """ If `a` is `None` then `b` is `str` """ a: Optional[int] b: Optional[str] # Some other things def someTypeGuard(my_obj: MyClass) -> ???: return my_obj.a is not None  I'm thinking it might be necessary for me to implement something to do with square brackets in type hints, but I really don't know where to start on this.

Answer: TypeGuard annotations can be used to annotate subclasses of a class. If parameter types are specified for those classes, then MyPy will recognise the type narrowing operation successfully. class MyClass: a: Optional[int] b: Optional[str] # Some other things # Two hidden classes for the different types class _MyClassInt(MyClass): a: int b: None class _MyClassStr(MyClass): a: None b: str def my_class_has_a(my_obj: MyClass) -> TypeGuard[_MyClassInt]: """Check if my_obj's `a` property is NOT `None`""" return my_obj.a is not None def my_class_has_b(my_obj: MyClass) -> TypeGuard[_MyClassStr]: """Check if my_obj's `b` property is NOT `None`""" return my_obj.b is not None  Sadly failure to narrow to one type doesn't automatically narrow to the other type, and I can't find an easy way to do this other than an assert my_class_has_b(obj) in your else block. Even still this seems to be the best solution.
Question: When sending a request to a specific URL I get an SSL error and I am not sure why. First please see the error message I am presented with:  requests.exceptions.SSLError: HTTPSConnectionPool(host='dicmedia.korean.go.kr', port=443): Max retries exceeded with url: /multimedia/naver/2016/40000/35000/14470_byeon-gyeong.wav (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:997)')))  I already tried:  https://github.com/Unbabel/COMET/issues/29 (This seems to be related with an internal update Python received relating to the use of specific SSL certificates (not an expert here) Downloading the certificate in question and directly linking to it with verify="private/etc/ssl/certs"  I am honestly at loss why I receive this error. As the error message itself indicates it seems that the server in question could get my local certificates somehow. The script worked until a week before. I did not update Python before then. Right now I use python 3.10.2 downloaded from the official website. I don't want to set verify=False as this just skips the verification process and leaves me vulnerable as numerous people already pointed out at different questions. Besides that it really bothers me that I can't resolve the error. Any help is much appreciated. See the specific request: import requests def request(url): response = requests.get(url, verify="/private/etc/ssl/certs") print(response) request("https://dicmedia.korean.go.kr/multimedia/naver/2016/40000/35000/14470_byeon- gyeong.wav") 

Answer: The problem was that not all certificates needed were included in Python's cacert.pem file. To tackle this I downloaded the certifi module at first. As this didn't work out as well I suppose as certifi also missed the necessary certificates. But I suppose not all certificates in the certificate where missing. As answers to similar questions indicated as well mostly what is missing is not the entire chain, but only the intermediate certificates. After: 1. downloading the necessary certificates (see the lock symbol in your browser; if you're on OSX you need to drag and drop the big images of the certificates to your finder or desktop etc.), 2. converting them to .perm files and bundling them together: cat first_cert.pem second_cert.pem > combined_cert.pem  and 3. providing the specific path of the bundled certificates as indicated in my question: verify="private/etc/ssl/certs (you may of course choose a different file path). my request got accepted by the server. I guess my mistake when trying this solution was that I didn't download the entire chain at first, but only the last certificate. I really hope this helps someone else as a point of reference. What I am still dying to know though, is why the error popped up in the first place. I didn't change my script at all and use it on a regular basis, but suddenly got presented with said error. Was the reason that the server I tried to reach change its certificates?
Question: How do I compare two polars DataFrames for value equality? It appears that == is only true if the two tables are the same object: import polars as pl pl.DataFrame({"x": [1,2,3]}) == pl.DataFrame({"x": [1,2,3]}) # False 

Answer: It's the equals method of DataFrame: import polars as pl pl.DataFrame({"x": [1,2,3]}).frame_equal(pl.DataFrame({"x": [1,2,3]})) # True  Before version 0.19.16, it was called frame_equals.
Question: I am running the Black code formatter against a Python script however it doesn't reformat the line length for docstrings. For example, given the following code: def my_func(): """ This is a really long docstring. This is a really long docstring. This is a really long docstring. This is a really long docstring. This is a really long docstring. This is a really long docstring. """ return  When running Black against this script, the line length does not change. How can I ensure docstrings get formatted when running Black?

Answer: maintainer here! :wave: The short answer is no you cannot configure Black to fix line length issues in docstrings currently. It's not likely Black will split or merge lines in docstrings as it would be far too risky, structured data can and does exist in docstrings. While I would hope the added newlines wouldn't break the consumers it's still a valid concern. There's currently an open issue asking for this (although it also wants the line length limit for docstrings and strings to be 79) GH-2289, and specifically for docstrings GH-2865. You can also read GH-1713 which is about splitting comments (and likewise has mixed feelings from maintainers). For the time being, perhaps you can look into https://github.com/PyCQA/docformatter which does seem to wrap docstrings (see the --wrap-descriptions and --wrap-summaries options)  P.S. if you're curious whether we'll add a flag to split docstrings or comments, it's once again unlikely since we seek to minimize formatting configurability. Especially as the pre-existing flags only disable certain elements of Black's style (barring --line-length which exists as there's no real consensus what it should be). Feel free to state your arguments in the linked issues tho!
Question: I use the IPython shell fairly often and have just started to notice it giving me strange autocomplete suggestions without any prompting from me. In this example, I just typed "im" and it suggests importing matplotlib?  This is very strange for several reasons: I've never seen this kind of grayed out code suggestion before that appears just as I type without the need to press tab or anything like that, the suggestions seem to be very arbitrary (why would typing im mean I want to import matplotlib of all things) and sometimes the suggestions make so sense (image 2: it just asks me to run plt.show() even though I haven't plotted anything yet).  Any clues to what could be going on here?

Answer: Try this. import IPython terminal = IPython.get_ipython() terminal.pt_app.auto_suggest = None  https://github.com/ipython/ipython/issues/13451