eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I have the following problem: I have a csv-file with faulty values (strings instead of integers) in some rows. To remedy that, I read it into polars and filter the dataframe. To be able to read it, I have to set infer_schema_length = 0, since otherwise the read would fail. This reads every column as a string, though. How would I re-infer the data types/schema of the corrected dataframe? I'd like to try to avoid setting every column individually, as there are a lot. I unfortunately can't edit the csv itself. ids_df = pl.read_csv(dataset_path, infer_schema_length=0) filtered_df = ids_df.filter(~(pl.col("Label") == "Label")) filtered_df.dtypes [Utf8, Utf8, Utf8, Utf8, Utf8, Utf8, Utf8, Utf8, Utf8, Utf8, ...  Thanks for your help.

Answer: I don't think Polars has this funtionality yet, but I think I found a valid way to solve your problem: from io import BytesIO import polars as pl dataset_path = "./test_data.csv" ids_df = pl.read_csv(dataset_path, infer_schema_length=0) print("ids_df",ids_df) filtered_df = ids_df.filter(~(pl.col("Label") == "Label")) print("filtered_df", filtered_df) # Save data to memory as a IO stream bytes_io = BytesIO() filtered_df.write_csv(bytes_io) # Read from IO stream with infer_schema_lenth != 0 new_df = pl.read_csv(bytes_io) print("new_df", new_df) bytes_io.close()
Question: Suppose I have a list of keywords and an input string labelled section, I want my code to find those keywords within section and put them inside [] square brackets. However, my keywords sometimes overlap with each other. keywords = ["alpha", "alpha beta", "alpha beta charlie", "alpha beta charlie delta"] To fix that I sorted them by length so that the longer keywords would be prioritized. However, when I run my code, sometimes I would get double or nested brackets (I assume it is because it still detects those as valid keywords) I tried this: import re keywords = ["alpha", "alpha beta", "alpha beta charlie", "alpha beta charlie delta"] keywords.sort(key=len, reverse=True) section = "alpha alpha beta alpha beta charlie alpha beta charlie delta" section = section.replace("'", "’").replace("\"", "”") section_lines = section.split('\n') for i, line in enumerate(section_lines): if not line.startswith('#'): section_lines[i] = re.sub(r'-',' ',line) for x in range(4): section_lines[i] = re.sub(r'\b' + f"{keywords[x]}" + r'\b', f"[{keywords[x]}]", section_lines[i], flags=re.IGNORECASE) section_lines[i] = section_lines[i].replace("[[", "[").replace("]]", "]") section = '\n'.join(section_lines) section = section.replace(" "," ").replace(" "," ") print(section)  Do not mind the line splits, it is for another part of to handle multiple lines. I wanted: [alpha] [alpha beta] [alpha beta charlie] [alpha beta charlie delta] but instead I got: [alpha] [alpha] beta] [alpha] beta] charlie] [alpha] beta] charlie] delta]

Answer: You can join the sorted keywords into an alternation pattern instead of substituting the string with different keywords 4 times, each time potentially substituting the replacement string from the previous iteration: import re keywords = ["alpha", "alpha beta", "alpha beta charlie", "alpha beta charlie delta"] keywords.sort(key=len, reverse=True) section = "alpha alpha beta alpha beta charlie alpha beta charlie delta" print(re.sub(rf"\b({'|'.join(keywords)})\b", r'[\1]', section))  This outpouts: [alpha] [alpha beta] [alpha beta charlie] [alpha beta charlie delta]
Question: I am trying to take mean of each group and trying to assign those to a new column in another dataframe but the first group's mean value is populating across all groups. Below is my dataframe df1 level value CF 5 CF 4 CF 6 EL 2 EL 3 EL 1 EF 4 EF 3 EF 6  I am taking the mean of each group and saving it to a new column in another dataframe df2. df2['value'] = df1.groupby(['level'])['value'].transform('mean')  But this is giving me below result level value CF 5.0 EL 5.0 EF 5.0  which should actually be level value CF 5.0 EL 2.0 EF 4.333333  I get expected result if I am not saving the values to new columnn. I am not sure if this is correct way of assigning group values to new column.

Answer: You should not use groupby.transform, this is mostly useful when you want to assign to the same dataframe. Here you need to map: df2['value'] = df2['level'].map(df1.groupby(['level'])['value'].mean())
Question: I have a series of subdirectories with files in them: /cars/ford/escape/sedan/ /cars/ford/escape/coupe/ /cars/ford/edge/sedan/ /cars/ferrari/testarossa/ /cars/kia/soul/coupe/  etcetera. I'd like to get all of these terminal subdirectory paths from the root, /cars/, using glob (within Python), but not include any of the files within them, nor any parents of a subdirectory. Each one contains only files, no further subdirectories. I tried using glob("**/"), but that also returns /cars/ford/, /cars/ford/escape/, /cars/ford/edge, /cars/ferrari/, etc. I do not want these. I also tried using rglob("*/") but that also returns all files inside the terminal subdirectories. I can get what I need by just globbing the files and making a set out of their parents, but I feel like there must be an elegant solution to this from the glob side of things. Unfortunately I can't seem to find the proper search terms to discover it. Thanks!

Answer: glob is the wrong tool for this job: traditional POSIX-y glob expressions don't support any kind of negative assertion (extglobs do, but it's still a restrictive kind of support -- making assertions about an individual name, not what does or doesn't exist on the same filesystem -- that doesn't apply to your use case, and Python doesn't support them anyhow). os.walk() and its newer children are better suited. Assuming you're on a new enough Python to support pathlib.Path.walk(): import pathlib def terminal_dirs(parent): for root, dirs, files in pathlib.Path(parent).walk(): if not dirs: yield root   For older versions of Python, os.walk() can be used similarly: import os def terminal_dirs(parent): for dirpath, dirnames, filenames in os.walk(parent): if not dirnames: yield dirpath   Both of these can of course be collapsed to one-liners if in a rush: result = [ r for (r,d,f) in os.walk('/cars') if not d ]
Question: After updating my OpenAI package to version 1.1.1, I got this error when trying to read the ChatGPT API response:  'ChatCompletion' object is not subscriptable  Here is my code: messages = [ {"role": "system", "content": '''You answer question about some service''' }, {"role": "user", "content": 'The user question is ...'}, ] response = client.chat.completions.create( model=model, messages=messages, temperature=0 ) response_message = response["choices"][0]["message"]["content"]  How can I resolve this error?

Answer: In the latest OpenAI package the response.choices object type is changed and in this way you must read the response: print(response.choices[0].message.content)  The complete working code: from openai import OpenAI client = OpenAI(api_key='YourKey') GPT_MODEL = "gpt-4-1106-preview" #"gpt-3.5-turbo-1106" messages = [ {"role": "system", "content": 'You answer question about Web services.' }, {"role": "user", "content": 'the user message'}, ] response = client.chat.completions.create( model=GPT_MODEL, messages=messages, temperature=0 ) response_message = response.choices[0].message.content print(response_message )  See this example in the project README.
Question: I'm trying to locally test a function i've written, but keep getting the error "cannot import name 'cygrpc' from 'grpc._cython" which causes attempts to run the function to fail I've made sure i'm running python 3.9 for compatibility, and the most recent version of azure functions core tools, as specified on the microsoft troubleshooting web page. Does anyone know of another work around for this?

Answer: What worked for me was running func init in the project (I'd cloned the project to a new machine and not run this step). After this it worked fine.
Question: This: df = (polars .DataFrame(dict( j=['2023-01-02 03:04:05.111', '2023-01-08 05:04:03.789'], k=['2023-01-02 03:04:05.222', '2023-01-02 03:04:05.456'], )) .select( polars.col('j').str.to_datetime(), polars.col('k').str.to_datetime(), ) .with_columns( l=polars.col('k') - polars.col('j'), ) ) print(df)  produces:  j (datetime[μs]) k (datetime[μs]) l (duration[μs]) 2023-01-02 03:04:05.111 2023-01-02 03:04:05.222 111ms 2023-01-08 05:04:03.789 2023-01-02 03:04:05.456 -6d -1h -59m -58s -333ms shape: (2, 3)  Doing this afterwards: print(df .select( polars.col('l').dt.to_string('%s.%f') ) )  raises: polars.exceptions.InvalidOperationError: `to_string` operation not supported for dtype `duration[μs]`  How would I go about formatting this to a string with a custom format like I can do with polars.Datetime?

Answer: It looks like you need to use pl.format and cobble together what you want with dt.total_seconds, dt.total_milliseconds, etc. These aren't intended to be used together so if you do it'll be (roughly) doubling (for 2 elements) the total time so, imo, best to use the most granular unit and do the math yourself since you've got to some some math anyways. This should work for the "%Ss%fms" case df.with_columns( z=pl.format("{}s{}ms", ((milli:=pl.col('l').dt.total_milliseconds()).abs().floordiv(1000))*milli.sign(), milli.abs().mod(1000)) ) shape: (2, 4) ┌─────────────────────────┬─────────────────────────┬──────────────────────────┬───────────────┐ │ j ┆ k ┆ l ┆ z │ │ --- ┆ --- ┆ --- ┆ --- │ │ datetime[μs] ┆ datetime[μs] ┆ duration[μs] ┆ str │ ╞═════════════════════════╪═════════════════════════╪══════════════════════════╪═══════════════╡ │ 2023-01-02 03:04:05.111 ┆ 2023-01-02 03:04:05.222 ┆ 111ms ┆ 0s111ms │ │ 2023-01-08 05:04:03.789 ┆ 2023-01-02 03:04:05.456 ┆ -6d -1h -59m -58s -333ms ┆ -525598s333ms │ └─────────────────────────┴─────────────────────────┴──────────────────────────┴───────────────┘
Question: Working on scripts to connect to AWS and recently started getting this error when I try to install a python module or execute a script I get the following error: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1129) It appears I have defined the certificate in the environment path so not sure what else to use to troubleshoot the issue.

Answer: Try this in the command line: pip install pip-system-certs  I've been struggling with the CERTIFICATE_VERIFY_FAILED error as well when using the requests module. I tried installing certifi but that didn't help. The only solution to my problem was to install pip-system-certs. For some reason that allowed requests to access a local certificate. Also: requests.get(url, verify=False) is not recommended for production environment because you basically turn safety off.
Question: I'm currently building a RAG (Retrieval Augmented Generation) Chatbot in Streamlit that queries my own data from a Postgres database and provides it as context for GPT 3.5 to answer questions about my own data. I have got the basics working already (frontend and backend). Now I also want to display the sources used nicely. I already finished the backend part. It returns a list of dictionaries with the following format: {"id": 1, "file_name": "my_file.pdf", "url_name": "", "content": "my content", "origin": "pdf"}  Some documents are actual files and therefore the "url_name" is "" and sometimes the source is an actual website. In that case "url_name" would be: "www.my_website.com" In order to display websources nicely I found the streamlit-extras library which has a nice feature to display mentions: from streamlit_extras.mention import mention mention(label=f"{file_name}", url=url_name)  It displays a clickable text with an icon. The url opens in another tab:  Now this does not really work for my non url documents. In that case I need something to show the actual text of the document. But I don't want to do it inline. I want something like this:  The Microsoft Azure App chatbot example provides the user the option to click on a source which will then show the content of the document on the right side of the page. I want to copy this behaviour by providing a selecatable text element that will expand the right sidebar in streamlit and display the text within this sidebar. Unfortunately I have no idea how I can accomplish this as there is no way to make a selectable text that can expand the sidebar and fill it with content as far as I know.

Answer: The simplest way is to serve the PDF via Flask/FastAPI and then use its link in your mention(label=f"{file_name}", url=url_name). The good news here is that most browsers will display the PDF on a specific page when it is provided. For example, if you go to https://arxiv.org/pdf/2401.00107.pdf#page=4, you will automatically land on page 4. So, in the simplest implementation, your url_name would be something like this: http://yourdomain.com/path_to_pdfs/{source["file_name"]}#page={source["page"]}. Now, if you want to display the PDF in your Streamlit app, you would need to embed it somehow with that link. You can use an HTML iframe as mentioned here: https://discuss.streamlit.io/t/rendering-pdf-on-ui/13505/21. Another option is to use st.components.v1.iframe: https://discuss.streamlit.io/t/aligning-st-components-v1-iframe/35885/4.
Question: Importing Polars polars==0.19.7 makes my kernel crash logs : import polars  The Kernel crashed while executing code in the the current cell or a previous cell. Please review the code in the cell(s) to identify a possible cause of the failure. Click here for more info. View Jupyter log for further details.  15:51:25.818 [info] Restarted 58cbe489-1e15-4ee8-a740-73c20717dff4 15:51:30.344 [info] Handle Execution of Cells 0 for ~/dev/playground/test.ipynb 15:51:30.352 [info] Kernel acknowledged execution of cell 0 @ 1699973490351 15:51:30.447 [error] Disposing session as kernel process died ExitCode: undefined, Reason: 15:51:30.448 [info] Dispose Kernel process 6740. 15:51:30.462 [info] End cell 0 execution @ undefined, started @ 1699973490351, elapsed time = -1699973490.351s  Would any one know what is causing this ?

Answer: have you tried https://github.com/pola-rs/polars/issues/12292 Do you want Polars to run on an old CPU (e.g. dating from before 2011), or on an x86-64 build of Python on Apple Silicon under Rosetta? Install pip install polars-lts-cpu. This version of Polars is compiled without AVX target features. "Celeron" and "Pentium" processors won't have AVX (even after 2011) https://www.ikmultimedia.com/faq/?id=1254 lscpu will tell you your processor family on ubuntu
Question: I'm using the PyPi code ( https://pypi.org/project/SpeechRecognition/)  cleaned up to use only Google Cloud SpeechRecognition. Google Json Credentials in shell's environment, and working.  I've enabled the Cloud Speech-to-Text API, got the Json credentials, and the service calls ARE hitting the API. The Microphone is fine, and the recording bit happens quickly. However, its taking fully 80 seconds to perform the API call! I've monitored the network traffic, and I can see that the API connection kinda sits idle for 78 seconds, and then TX/RXs really fast in the final 2 seconds. How can I speed this up? Can it be slow-authentication handshake that I might mend? MORE INFORMATION: My application performs 3 API calls: Google Speech-to-text | Google Translate text-to-text | Google text-to-Speech. Those API Calls ALWAYS take 80 seconds , 20s & 80s respectively. Thanks a mil! The delay happens on the 2nd last line here: print("0 seconds") try: print("Google Cloud Speech thinks you said " + r.recognize_google_cloud(audio) print("80th second") 

Answer: Problem solved! It was the SSL certs; bouquets to @Dean Van Greune & @VonC My fibre router (Sagemcom) was blocking the Pi's SSL certs, or forcing it to a different port, creating massive delays. I remember solving the same problem for JavaMail TLS a while back, and wanting to take the bat to the router ("Office Space"-style). Swapping to the hotspot on my phone, all this is working faster than the lightiest lightning now. Thanks for your help & suggestions guys!!!
Question: The typing.TypeVar class allows one to specify reusable type variables. With Python 3.12 / PEP 695, one can define a class A/B with type variable T like this: class A[T]: ... class B[T]: ...  Beforehand, with Python 3.11, you would do it like this: from typing import TypeVar, Generic T = TypeVar("T") class A(Generic[T]): ... class B(Generic[T]): ...  For the first example, the T is defined in the class scope, so they do not relate to each other. Is there any difference to the second 'old' example? Or: Is there any connection between the two classes A and B?

Answer: There is no such connection from the typechecker's POV. TypeVar is declared outside of class scope just because it's convenient to do so, it does not imply any relationships between its users. Type variable is bound in the following scopes:  Class scope - if a class inherits from Generic (or parametrized Protocol, or other generic class), type variable that parametrizes its parents is bound to class scope. All occurrences of this type variable are resolved to the same type in same context. Function/method scope - if a function includes a type variable as part of its parameter(s) or return type, this type variable resolves in function body to the same type as in signature.  No other binding is taking place. This is explained in detail in PEP 484 (link to relevant section). However, there may be clean semantic relationship (and it was one of my arguments against PEP695 - a battle I, unfortunately, lost). Consider django-stubs (permalink to the declaration): # __set__ value type _ST = TypeVar("_ST", contravariant=True) # __get__ return type _GT = TypeVar("_GT", covariant=True) class Field(RegisterLookupMixin, Generic[_ST, _GT]): ... # omitted (there are 150+ lines here in fact) class IntegerField(Field[_ST, _GT]): ... class PositiveIntegerRelDbTypeMixin: ... class SmallIntegerField(IntegerField[_ST, _GT]): ... class BigIntegerField(IntegerField[_ST, _GT]): ... class PositiveIntegerField(PositiveIntegerRelDbTypeMixin, IntegerField[_ST, _GT]): ... class PositiveSmallIntegerField(PositiveIntegerRelDbTypeMixin, SmallIntegerField[_ST, _GT]): ... class PositiveBigIntegerField(PositiveIntegerRelDbTypeMixin, BigIntegerField[_ST, _GT]): ...  These fields do not have to resolve to the same type variables, obviously. However, _ST and _GT have a semantic meaning, which is same for all of them. This does not influence type checking, but helps those who read this code. If _GT and _ST were defined for each class separately, then, in addition to unreadable and unpleasantly looking syntax, we'd have to either repeat comments near every class or document that once in module docstring and add one extra lookup step to demystify their meaning. Using longer forms like _SetterType and _GetterType would remove need for explanation, but also make the already long-ish signatures longer. Additionally, explanation of how Field typing works is still necessary (there are other caveats). Now it's explained in a docstring that comes right after typevar declaration, but with PEP695 style it'd be written far away.
Question: I am trying to backtest a momentum strategy using Backtesting.py. I've gathered the data and computed indicator values using pandas_ta. I've defined short and long trading conditions. Now I just need Backtesting.py to run a backtest so that I can determine the performance of my strategy on historical data. I am expecting stats from bt.run() to show me values instead of nan and 0. I am working using this notebook: https://github.com/kbs-code/algo_trader/blob/master/backtests/backtestingdotpy_SPY_weekly_vortex.ipynb EDIT: The name of this notebook is now https://github.com/kbs-code/algo_trader/blob/master/backtests/debugging_example_backtestingdotpy.ipynb Thanks

Answer: The backtester must use "self.data.foo" and not "self.foo" in order to use values inside any column. Please see the updated notebook with troubleshooting proof.
Question: I have two dataclasses: Msg and Field. Msg has a field fields of type list[Field]. I want to assign something to a field of each Field after they have all been initialized which is more or less their relative index in the fields list. However, when I add a __post_init__(self) method to the Msg dataclass, the fields list is empty, so I can't update the indices. from dataclasses import dataclass from ruamel.yaml import YAML @dataclass class Msg: id: int desc: str fields: list[Field] def __post_init__(self) -> None: idx: int = 0 for field in self.fields: # why is this empty?? field.index = idx idx += field.size @dataclass class Field: id: int name: str units: str size: int index: int = -1 y = YAML() y.register_class(Msg) y.register_class(Field) msg: Msg = y.load("""\ !Msg id: 1 desc: status fields: - !Field id: 1 name: Temp units: degC size: 2 """) assert(msg.fields[0].index != -1) # fails :(  Why is this? How is the Msg being initialized without fields being initialized? Is there any way to do what I am trying to do using the class system? I am using Python 3.11 with ruamel.yaml 0.18.5 on MacOS.

Answer: By default, object serializers such as YAML and pickle have no idea what to do with the attribute mapping for a user-defined object other than to assign the mapping directly to the object's attribute dictionary as-is. This is why you can define a __setstate__ method for your class, so that ruamel.yaml's object constructor knows in this case to call the __init__ method with the mapping unpacked as arguments, which in turn calls __post_init__ for post-initialization: @dataclass class Msg: id: int desc: str fields: list[Field] def __post_init__(self) -> None: idx: int = 0 for field in self.fields: field.index = idx idx += field.size def __setstate__(self, state): self.__init__(**state)  Demo: https://replit.com/@blhsing1/PowerfulThoseJavadocs