eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I am trying to write a function to log dataclasses I would like to get the name of all fields in the dataclass and print the value to each (similar to how you might write a function to print a dictionary) i.e. import dataclasses @dataclasses.dataclass class Test: a: str = "a value" b: str = "b value" test = Test() def print_data_class(dataclass_instance): fields = ... # get dataclass fields for field in fields: print(f"{field.name}: {field.value}") print_data_class(test)  Desired output: "a": "a value" "b": "b value"  However I haven't been able to find how to get the fields of a dataclass, does anyone know how this could be done?

Answer: This example shows only a name, type and value, however, __dataclass_fields__ is a dict of Field objects, each containing information such as name, type, default value, etc. Using dataclasses.fields() Using dataclasses.fields() you can access fields you defined in your dataclass. fields = dataclasses.fields(dataclass_instance)  Using inspect.getmembers() Using inspect.getmembers() you can access all fields in your dataclass. members = inspect.getmembers(type(dataclass_instance)) fields = list(dict(members)['__dataclass_fields__'].values())  Complete code solution import dataclasses import inspect @dataclasses.dataclass class Test: a: str = "a value" b: str = "b value" def print_data_class(dataclass_instance): # option 1: fields fields = dataclasses.fields(dataclass_instance) # option 2: inspect members = inspect.getmembers(type(dataclass_instance)) fields = list(dict(members)['__dataclass_fields__'].values()) for v in fields: print(f'{v.name}: ({v.type.__name__}) = {getattr(dataclass_instance, v.name)}') print_data_class(Test()) # a: (str) = a value # b: (str) = b value print_data_class(Test(a="1", b="2")) # a: (str) = 1 # b: (str) = 2
Question: Currently when I try to retrieve date from a polars datetime column, I have to write something similar to: import polars as pl import datetime as dt df = pl.DataFrame({ 'time': [dt.datetime.now()] }) df = df.with_columns( pl.col("time").map_elements(lambda x: x.date()).alias("date") )  shape: (1, 2) ┌────────────────────────────┬────────────┐ │ time ┆ date │ │ --- ┆ --- │ │ datetime[μs] ┆ date │ ╞════════════════════════════╪════════════╡ │ 2024-07-20 11:41:04.265539 ┆ 2024-07-20 │ └────────────────────────────┴────────────┘  Is there a different way, something closer to: pl.col("time").dt.date().alias("date") 

Answer: You can use .dt.date() import datetime import polars as pl df = pl.DataFrame({ "time": [datetime.datetime.now()] }) df.with_columns( pl.col("time").dt.date().alias("date") )  shape: (1, 2) ┌────────────────────────────┬────────────┐ │ time ┆ date │ │ --- ┆ --- │ │ datetime[μs] ┆ date │ ╞════════════════════════════╪════════════╡ │ 2024-07-21 16:17:41.489579 ┆ 2024-07-21 │ └────────────────────────────┴────────────┘
Question: Is it possible to shuffle two 2D tensors in PyTorch by their rows, but maintain the same order for both? I know you can shuffle a 2D tensor by rows with the following code: a=a[torch.randperm(a.size()[0])]  To elaborate: If I had 2 tensors a = torch.tensor([[1, 1, 1, 1, 1], [2, 2, 2, 2, 2], [3, 3, 3, 3, 3]]) b = torch.tensor([[4, 4, 4, 4, 4], [5, 5, 5, 5, 5], [6, 6, 6, 6, 6]])  And ran them through some function/block of code to shuffle randomly but maintain correlation and produce something like the following a = torch.tensor([[2, 2, 2, 2, 2], [1, 1, 1, 1, 1], [3, 3, 3, 3, 3]]) b = torch.tensor([[5, 5, 5, 5, 5], [4, 4, 4, 4, 4], [6, 6, 6, 6, 6]])  My current solution is converting to a list, using the random.shuffle() function like below. a_list = a.tolist() b_list = b.tolist() temp_list = list(zip(a_list , b_list )) random.shuffle(temp_list) # Shuffle a_temp, b_temp = zip(*temp_list) a_list, b_list = list(a_temp), list(b_temp) # Convert back to tensors a = torch.tensor(a_list) b = torch.tensor(b_list)  This takes quite a while and was wondering if there is a better way.

Answer: You can use the function torch.randperm to get a set of indices that act as a random permutation. The following is a small example of getting a random permutation, then applying it to both the a and b tensors: indices = torch.randperm(a.size()[0]) a=a[indices] b=b[indices]
Question: I have read and tried a lot but am still struggeling with correctly configuring IntelliSense, ie, VSCode's autocomplete for my Python projects. It works fine within a single Python project. But I have a workspace with two of my projects open at the same time since one of them is imported and used inside the other. For simplicity, let's call them myProjectA and myProjectB. Inside myProjectA, I get correct autocomplete (incl. suggestions and auto import) for myProjectA but not for myProjectB, which is used inside A. How can I configure VSCode to show auto suggestions of myProjectB in myProjectA? I have tried adding "python.autoComplete.extraPaths": [ "myProjectB" ]  to my config, but it doesn't change anything. Do I need to provide the full path to myProjectB? Or do I need to configure something else?

Answer: it wasn't working for me until i added the path to "python.analysis.extraPaths" as well: "python.analysis.extraPaths": [ "path\\to\\directory" ], "python.autoComplete.extraPaths": [ "path\\to\\directory" ]
Question: I have this model: class Text(BaseModel): id: str text: str = None class TextsRequest(BaseModel): data: list[Text] n_processes: Union[int, None]  So I want to be able to take requests like: {"data": ["id": "1", "text": "The text 1"], "n_processes": 8}  and {"data": ["id": "1", "text": "The text 1"]}.  Right now in the second case I get {'data': [{'id': '1', 'text': 'The text 1'}], 'n_processes': None}  using this code: app = FastAPI() @app.post("/make_post/", response_model_exclude_none=True) async def create_graph(request: TextsRequest): input_data = jsonable_encoder(request)  So how can I exclude n_processes here?

Answer: Since Pydantic >= 2.0 deprecates model.dict() use model.model_dump(...) instead.  You can use exclude_none param of Pydantic's model.dict(...): class Text(BaseModel): id: str text: str = None class TextsRequest(BaseModel): data: list[Text] n_processes: Optional[int] request = TextsRequest(**{"data": [{"id": "1", "text": "The text 1"}]}) print(request.dict(exclude_none=True))  Output: {'data': [{'id': '1', 'text': 'The text 1'}]}  Also, it's more idiomatic to write Optional[int] instead of Union[int, None].
Question: How would I use TypeVarTuple for this example? T = TypeVar("T") Ts = TypeVarTuple("Ts") @dataclass class S(Generic[T]): data: T def data_from_s(*structs: ??) -> ??: return tuple(x.data for x in structs) a = data_from_s(S(1), S("3")) # is type tuple[int, str] 

Answer: I don't see any way to do this with the current spec. The main issue I see is that TypeVarTuple does not support bounds. You can't constrain the types referred to by Ts to be bounded to S. You need to translate somehow tuple[S[T1], S[T2], ...] -> tuple[T1, T2, ...], but you have no way to know that the types contained by Ts are specializations of S or that types themselves are generic with further parameterization.  Without using TypeVarTuple, your goal can be accomplished to some extent with a pattern like the following, using overload to handle subsets of the signature for differing amounts of arguments. I also use an ending / in the overloads to prevent usage of named arguments (forcing positional args to be used), which allows the overloads to match the real method definition. Obviously, this pattern becomes awkward as you add more ranges of arguments, but in some cases it can be a nice escape hatch. from dataclasses import dataclass from typing import Any, Generic, TypeVar, assert_type, overload T = TypeVar("T") @dataclass class S(Generic[T]): data: T ... T1 = TypeVar("T1") T2 = TypeVar("T2") T3 = TypeVar("T3") @overload def data_from_s(s1: S[T1], /) -> tuple[T1]: ... @overload def data_from_s(s1: S[T1], s2: S[T2], /) -> tuple[T1, T2]: ... @overload def data_from_s(s1: S[T1], s2: S[T2], s3: S[T3], /) -> tuple[T1, T2, T3]: ... def data_from_s(*structs: S[Any]) -> tuple[Any, ...]: return tuple(x.data for x in structs)  Which will pass this test: assert_type( data_from_s(S(1)), tuple[int] ) assert_type( data_from_s(S(1), S("3")), tuple[int, str] ) assert_type( data_from_s(S(1), S("3"), S(3.9)), tuple[int, str, float] )
Question: I am trying to index word embedding vectors to Elasticsearch V8 ann dense_vector dot_product. I can successfully index vec to cosine, so I converted it to unit vector with numpy for dot_product.  unit_vector = vec / np.linalg.norm(vec)  but I get an 400 error saying like this. The [dot_product] similarity can only be used with unit-length vectors. Preview of invalid vector: [-0.0038341882, -0.1564709, 0.08771773, -0.14555556, -0.07952896, ...]  Am I missing something?

Answer: I was confronted with the exact same problem and I found a solution after much experimentation. In my case, when indexing lots of embeddings to Elasticsearch (dense_vector with similarity parameter set to dot_product), most of them got indexed properly and a small percentage of them failed with The [dot_product] similarity can only be used with unit-length vectors. I found after intensive testing that the problem was that the unit vectors I was working with were of numerical types np.float16 and this was causing the error. Working with np.float32 as a numerical type in my workflow for my unit vectors solved the issue.
Question: I'm trying to to get OpenCV with CUDA to be used in Python open-cv on Arch Linux, but I'm not sure how to link it. Arch provides a package opencv-cuda, which provides these files. Guides I've found said to link the python cv2.so to the one provided, but the package doesn't provide that. My python site_packages has cv2.abi3.so in it, and I've tried linking that to core.so and cvv.so to no avail. Do I need to build it differently to support Python? Or is there another step I'm missing?

Answer: On Arch, opencv-cuda provides opencv=4.6.0, but you still need the python bindings. Fortunately though, installing python-opencv after installling opencv-cuda works, since it leverages it. I just set up my Python virtual environment to allow system site packages (python -m venv .venv --system-site-packages), and it works like a charm! Neural net image detection runs ~300% as fast now.
Question: I want to express that the first parameter is a "list" of the second parameter, and that the result has the same type as the second parameter. This mysum (ie. not the standard lib sum) should work equally well with int/float/str/list, and any other type that supports +=. Naively: def mysum(lst: list[T], start: T) -> T: x = start for item in lst: x += item return x  which produces: (dev311) go|c:\srv\tmp> mypy sumtype.py sumtype.py:26: error: Unsupported left operand type for + ("T") sumtype.py:35: error: Cannot infer type argument 1 of "mysum" Found 2 errors in 1 file (checked 1 source file)  Second attempt: from typing import Iterable, Protocol, TypeVar T = TypeVar('T') class Addable(Protocol[T]): def __add__(self, other: T) -> T: ... class RAddable(Protocol[T]): def __radd__(self, other: T) -> T: ... def mysum(lst: Iterable[Addable|RAddable], start: Addable) -> Addable: x = start for item in lst: x += item return x  however, this doesn't place any restrictions on the "list" items and start being the same type, so this typechecks: class Foo: def __radd__(self, other: int) -> int: return other + 42 mysum([Foo()], []) # <=== should ideally fail since Foo() and [] are not the same type  and fails with a type-error at runtime: (dev311) go|c:\srv\tmp> python sumtype.py Traceback (most recent call last): File "c:\srv\tmp\sumtype.py", line 27, in <module> mysum([Foo()], []) File "c:\srv\tmp\sumtype.py", line 18, in mysum x += item File "c:\srv\tmp\sumtype.py", line 24, in __radd__ return other + 42 ~~~~~~^~~~ TypeError: can only concatenate list (not "int") to list  I'm assuming it can't be this difficult to type-annotate a generic function, so what am I missing..?

Answer: You can make a generic type bound to a protocol that supports __add__ instead:  Type variables can be bound to concrete types, abstract types (ABCs or protocols), and even unions of types  from typing import TypeVar, Protocol T = TypeVar('T', bound='Addable') class Addable(Protocol): def __add__(self: T, other: T) -> T: ... def mysum(lst: list[T], start: T) -> T: x = start for item in lst: x += item return x  Demo: https://mypy-play.net/?gist=ecf4031f21fb81e75d6e6f75d16f3911
Question: I have a dataset where I would like to add or append rows with values listed in dictionary (if these values are missing from original dataset) Data ID Date Type Cost Alpha Q1 2022 ok 1 Alpha Q2 2022 ok 1 Alpha Q3 2022 hi 1 Alpha Q4 2022 hi 2  Desired ID Date Type Cost Alpha Q1 2022 ok 1 Alpha Q2 2022 ok 1 Alpha Q3 2022 hi 1 Alpha Q4 2022 hi 2 Gamma Q1 2022 0 Theta Q1 2022 0  Doing I am using the script below, however, this is not appending, but only maps the value if date matches. Any suggestion is appreciated #values = {'Alpha': 'Q1 2022', 'Gamma':' Q1 2022', 'Theta': 'Q1 2022'} df['ID']=out['Date'].map({'Alpha': 'Q1 2022', 'Gamma':' Q1 2022', 'Theta': 'Q1 2022' }) df1 = df1.merge(df, how='left').fillna({'Cost': 0}) 

Answer: Let us create a delta dataframe from the items of dictionary, then do a outer merge to append distinct rows delta = pd.DataFrame(values.items(), columns=['ID', 'Date']) df.merge(delta, how='outer')    ID Date Type Cost 0 Alpha Q1 2022 ok 1.0 1 Alpha Q2 2022 ok 1.0 2 Alpha Q3 2022 hi 1.0 3 Alpha Q4 2022 hi 2.0 4 Gamma Q1 2022 NaN NaN 5 Theta Q1 2022 NaN NaN
Question: I wonder how i can transform Spark dataframe to Polars dataframe. Let's say i have this code on PySpark: df = spark.sql('''select * from tmp''')  I can easily transform it to pandas dataframe using .toPandas. Is there something similar in polars, as I need to get a polars dataframe for further processing?

Answer: Context Pyspark uses arrow to convert to pandas. Polars is an abstraction over arrow memory. So we can hijack the API that spark uses internally to create the arrow data and use that to create the polars DataFrame. TLDR Given an spark context we can write: import pyarrow as pa import polars as pl sql_context = SQLContext(spark) data = [('James',[1, 2]),] spark_df = sql_context.createDataFrame(data=data, schema = ["name","properties"]) df = pl.from_arrow(pa.Table.from_batches(spark_df._collect_as_arrow())) print(df)  shape: (1, 2) ┌───────┬────────────┐ │ name ┆ properties │ │ --- ┆ --- │ │ str ┆ list[i64] │ ╞═══════╪════════════╡ │ James ┆ [1, 2] │ └───────┴────────────┘  Serialization steps This will actually be faster than the toPandas provided by spark itself, because it saves an extra copy. toPandas() will lead to this serialization/copy step: spark-memory -> arrow-memory -> pandas-memory With the query provided we have: spark-memory -> arrow/polars-memory
Question: I'm trying to connect to Azure SQL Database from Azure Synapse workspace Notebook using PySpark. Also I would like to use Active Directory integrated authentication. So what I've tried: jdbc_df = spark.read \ .format("com.microsoft.sqlserver.jdbc.spark") \ .option("url", "jdbc:sqlserver://my_server_name.database.windows.net:1433") \ .option("database","my_db_name") \ .option("dbtable", "my_table_or_query") \ .option("authentication", "ActiveDirectoryIntegrated") \ .option("encrypt", "true") \ .option("hostNameInCertificate", "*.database.windows.net") \ .load()  Also I've tried the same way but in different syntax jdbcUrl = "jdbc:sqlserver://my_server_name.database.windows.net:1433;database=my_db_name;encrypt=true;trustServerCertificate=false;hostNameInCertificate=*.database.windows.net;loginTimeout=30;Authentication=ActiveDirectoryIntegrated" pushdown_query = "SELECT col1 FROM my_table_name" connectionProperties = { "driver" : "com.microsoft.sqlserver.jdbc.SQLServerDriver" } df = spark.read.jdbc(url=jdbcUrl, table=pushdown_query) display(df)  And in both cases I get error  IllegalArgumentException: KrbException: Cannot locate default realm  What I'm doing wrong?

Answer: Finally I have found the solution! First of all there should be created working Linked service to Azure SQL database in your Synapse Analytics that uses Authentication type "System Assigned Managed Identity". Than you can reference it in your PySpark Notebook. And don't be confused that method getConnectionString is used to get access token - it really returns not connection string but token. jdbcUrl = "jdbc:sqlserver://my_server_name.database.windows.net:1433;database=my_db_name;encrypt=true;trustServerCertificate=false;hostNameInCertificate=*.database.windows.net;loginTimeout=30" token=TokenLibrary.getConnectionString("AzureSQLLinkedServiceName") pushdown_query = "(SELECT col1 FROM my_table_name) as tbl" connectionProperties = { "driver" : "com.microsoft.sqlserver.jdbc.SQLServerDriver", "accessToken" : token } df = spark.read.jdbc(url=jdbcUrl, table=pushdown_query, properties=connectionProperties) display(df)
Question: I use this spacy code to later apply it on my text, but i need the negative words to stay in the text like "not". nlp = spacy.load("en_core_web_sm") def my_tokenizer(sentence): return [token.lemma_ for token in tqdm(nlp(sentence.lower()), leave = False) if token.is_stop == False and token.is_alpha == True and token.lemma_ ]  Whit this when i apply i get this as a result : [hello, earphone, work]  However the original sentence was hello,my earphones are still not working.  So, i would like to see the following sentence: [earphone, still, not, work] Thank you

Answer: "not" is actually a stop word and in your code if a token is removed if it's a stopword. You can see this either by looking at the list of Spacy stopwords  "not" in spacy.lang.en.stop_words.STOP_WORDS  or by looping over the tokens of your doc object for tok in nlp(text.lower()): print(tok.text, tok.is_stop, tok.lemma_) #hello False hello #, False , #my True my #earphones False earphone #are True be #still True still #not True not #working False work #. False .  Solution To solve this, you should remove the target words such as "not" from the list of stop_words. You can do it this way:  # spacy.lang.en.stop_words.STOP_WORDS.remove("not") # or for multiple words use this to_del_elements = {"not", "no"} nlp.Defaults.stop_words = nlp.Defaults.stop_words - to_del_elements  Then you can rerun your code and you'll get your expected results: import spacy #spacy.lang.en.stop_words.STOP_WORDS.remove("not") to_del_elements = {"not", "no"} nlp.Defaults.stop_words = nlp.Defaults.stop_words - to_del_elements nlp = spacy.load("en_core_web_sm") def my_tokenizer(sentence): return [token.lemma_ for token in tqdm(nlp(sentence.lower()), leave = False) if token.is_stop == False and token.is_alpha == True and token.lemma_ ] sentence = "hello,my earphones are still not working. no way they will work" results = my_tokenizer(sentence) print(results) #['hello', 'earphone', 'not', 'work', 'no', 'way', 'work']
Question: On Windows 10, python3.10, PySide6 (or PyQt6) QApplication crashes when calling QPainter.drawLine() . The terminal just displays :  Process finished with exit code -1073741819 (0xC0000005)  Please find below the code: import sys from PySide6.QtCore import QPoint, Qt from PySide6.QtGui import QColor, QPainter, QPen, QPixmap from PySide6.QtWidgets import QApplication, QLabel, QMainWindow # from PyQt6.QtCore import QPoint, Qt # from PyQt6.QtGui import QColor, QPainter, QPen, QPixmap # from PyQt6.QtWidgets import QApplication, QLabel, QMainWindow class MainWindow(QMainWindow): def __init__(self): super().__init__() self.label = QLabel() canvas = QPixmap(400, 300) canvas.fill(Qt.GlobalColor.white) self.label.setPixmap(canvas) self.setCentralWidget(self.label) self.draw_something() def draw_something(self): painter = QPainter(self.label.pixmap()) painter.drawLine(10, 10, 300, 200) # >=========== Crash Here painter.end() if __name__ == '__main__': app = QApplication(sys.argv) window = MainWindow() window.show() app.exec() 

Answer: This is caused by a slight (and not well documented) change in the API that happened starting with Qt5.15. Until Qt5, pixmap() returned a direct pointer to the current pixmap of the label, while in Qt6 it returns an implicit copy of the pixmap. The difference is highlighted only for the latest Qt5 documentation of the pixmap() property:  Previously, Qt provided a version of pixmap() which returned the pixmap by-pointer. That version is now deprecated. To maintain compatibility with old code, you can explicitly differentiate between the by-pointer function and the by-value function:  To Python developers that's not obvious, but to C++ it's clear by the change of the const QPixmap * (note the asterisk, meaning that it's a pointer) to a pure QPixmap type, meaning that the returned object is a new QPixmap object based on the current pixmap, and not a reference to the pixmap object currently set for the label. Now, the fact is that, conceptually speaking, we should not be able to directly "live draw" on the current pixmap of the label, because:  setPixmap() always creates a copy of the pixmap for the label; due to the point above, there is no point in sharing the same pixmap object among different labels, so, as a consequence, pixmap() should always return a copy of that object;  Previously, it was possible to directly paint on a QLabel's pixmap (while ensuring that update() was immediately called on the label). The current API instead requests an explicit call to setPixmap() after drawing. So, the solution is to create a reference to the pixmap as long as it's needed:  def draw_something(self): pm = self.label.pixmap() painter = QPainter(pm) painter.drawLine(10, 10, 300, 200) painter.end() self.label.setPixmap(pm)
Question: I'm trying to save Snakemake's own console output (not the logs generated by the individual jobs) to an arbitrary file while still having it written to stdout/stderr. (Unfortunately, my setup means I can't just use tee.)  It looks to me like Snakemake should provide that functionality, given it saves the log output to a default location. However, looking through the documentation, I couldn't find a parameter to easily change the output location for the log. From the code of snakemake.logging, I'm getting the impression the logfile's parent directory may be hardcoded, with the file just getting a timestamped name. Is there some bleedingly obvious way to configure what file Snakemake logs to that I'm overlooking?

Answer: The file name/path is hardcoded in setup_logfile. It's a hack, but one option is to copy the log file to the desired location using onsuccess/onerror (note that there is no oncompletion, so the log copying should probably apply to both cases): onsuccess: shell("cp -v {log} some_path_for_log_copy.log") onerror: shell("cp -v {log} some_path_for_log_copy.log")
Question: I am trying to generate random chinese names using Faker (Python), but it generates the names in chinese characters instead of pinyin. I found this :  and it show that it generates them in pinyin, while when I try the same code, it gives me only chinese characters. how to get the pinyin ??

Answer: fake.romanized_name() worked for me. I got lucky by looking through dir(fake). Doesn't seem to have a method for pinyin address that I can see...