eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I am using the polars library in python and have two data frames the look like this import polars as pl data1 = { 'A': [1, 2, 3], 'B': [4, 5, 6], 'C': [7, 8, 9] } df1 = pl.DataFrame(data1) data2 = { 'B': [4, 5, 6], 'C': [7, 8, 9], 'D': [1, 2, 3] } df2 = pl.DataFrame(data2) # the column B and C are same in both data frames # TODO: Join/Concat the data frames into one.  The data2 can vary some time it can have 2 common columns, some time it can have 1 common column and some times more. and The result should look like. I was wondering if there is any built function, or some kind of flag in a function that exists already in polars, that i can use. result = { 'A': [1, 2, 3], 'B': [4, 5, 6], 'C': [7, 8, 9], 'D': [1, 2, 3] }  I am not quite sure how to join or concat the polar data frames in order to achieve this.

Answer: You can concat with how="align" but the resulting column order differs. pl.concat([df1, df2], how="align")  shape: (3, 4) ┌─────┬─────┬─────┬─────┐ │ B ┆ C ┆ A ┆ D │ │ --- ┆ --- ┆ --- ┆ --- │ │ i64 ┆ i64 ┆ i64 ┆ i64 │ ╞═════╪═════╪═════╪═════╡ │ 4 ┆ 7 ┆ 1 ┆ 1 │ │ 5 ┆ 8 ┆ 2 ┆ 2 │ │ 6 ┆ 9 ┆ 3 ┆ 3 │ └─────┴─────┴─────┴─────┘  You can see how it is implemented here. It basically finds the common columns to use as on and "outer joins" all the frames together. dfs = [df1, df2] cols = pl.concat(pl.Series(df.columns) for df in dfs) join_cols = list(cols.filter(cols.is_duplicated()).unique()) result = dfs[0].join(dfs[1], how="outer", on=join_cols, suffix="") # only needed for more than 2 frames # for df in dfs[2:]: # result = result.join(df, how="outer", on=join_cols, suffix="") result = result.select(*cols.unique(maintain_order=True))  shape: (3, 4) ┌─────┬─────┬─────┬─────┐ │ A ┆ B ┆ C ┆ D │ │ --- ┆ --- ┆ --- ┆ --- │ │ i64 ┆ i64 ┆ i64 ┆ i64 │ ╞═════╪═════╪═════╪═════╡ │ 1 ┆ 4 ┆ 7 ┆ 1 │ │ 2 ┆ 5 ┆ 8 ┆ 2 │ │ 3 ┆ 6 ┆ 9 ┆ 3 │ └─────┴─────┴─────┴─────┘
Question: I have a setup like the following class decorator: def __init__(self, fn): self.fn = fn @staticmethod def wrap(fn): return decorator(fn) def __call__(self): print(f"decorator {self} function called") class A: @decorator.wrap def foo(self): print(f"Object {self} called") def __init__(self): self.boo = 'Boo'  How do I from the decorator object access boo variable?

Answer: @S.B's solution works but is not thread-safe since multiple instances of A may be calling the foo method at about the same time in different threads, overriding the decorator object's instance attribute before a call to the __call__ method originated from another instance is made. A thread-safe approach would be to return a wrapper function from the __get__ method with the instance available through closure: class decorator: def __init__(self, fn): self.fn = fn def __get__(self, instance, owner=None): def wrapper(*args, **kwargs): print(f'decorator {self} function called; boo = {instance.boo}') return self.fn(instance, *args, **kwargs) return wrapper class A: @decorator def foo(self): print(f"Object {self} called") def __init__(self): self.boo = 'Boo' A().foo()  This outputs: decorator <__main__.decorator object at 0x7f6ef3316c10> function called; boo = Boo Object <__main__.A object at 0x7f6ef3339580> called  Demo: Try it online!
Question: I want to do something pretty simple here: import a module from the local filesystem using databricks asset bundles. These are the relevant files: databricks.yml bundle: name: my_bundle workspace: host: XXX targets: dev: mode: development default: true resources: jobs: my_job: name: my_job tasks: - task_key: my_task existing_cluster_id: YYY spark_python_task: python_file: src/jobs/bronze/my_script.py  my_script.py from src.jobs.common import * if __name__ == "__main__": hello_world()  common.py def hello_world(): print("hello_world")  And the following folder structure: databricks.yml src/ ├── __init__.py └── jobs ├── __init__.py ├── bronze │ └── my_script.py └── common.py  I'm deploying this to my workspace + running it by using Databricks CLI v0.206.0 with the following commands: databricks bundle validate databricks bundle deploy databricks bundle run my_job  I'm getting issues to import my common.py module. I'm getting the classic ModuleNotFoundError: No module named 'src' error here. I've added the __init__.py files as I typically do when doing this locally, and tried the following variations: from src.jobs.common import * from jobs.common import * from common import * from ..common import *  I guess my issue is that I don't really know what the python path is here, since I'm deploying it on Databricks. How can I do something like this using databricks asset bundles?

Answer: I recently ran into a similar issue, albeit with notebook tasks, and came to the following resolution, adapted to your example file structure: In your databricks.yml file, pass an argument to your script via parameters: databricks.yml  resources: jobs: my_job: name: my_job tasks: - task_key: my_task existing_cluster_id: YYY spark_python_task: python_file: src/jobs/bronze/my_script.py parameters: ['/Workspace/${workspace.file_path}/src']  main.py  import sys bundle_src_path = sys.argv[1] sys.path.append(bundle_src_path) from src.jobs.common import *  Caveats:  As mentioned, I am using a notebook_task which allows me to pass in parameters that I can read using dbutils. I haven't tested the spark_python_task parameters passing above, but it appears similar and may at least be enough to get you into a working state. Databricks API reference The sys path technique is recommended by Databricks docs, though you may need to approach it differently based on your runtime version. This works well for non-production deployment targets. The parameter passed in for production would likely need to be modified (though I haven't made it that far myself yet)!
Question: I have a directory structure on my filesystem, like this: folder_to_scan/ important_file_a important_file_b important_folder_a/ important_file_c important_folder_b/ important_file_d useless_folder/ ...  I want to recursively scan through folder_to_scan/, and get all the file names. At the same time, I want to ignore useless_folder/, and anything under it. If I do something like this: path_to_search = Path("folder_to_scan") [pth for pth in path_to_search.rglob("*") if pth.is_file() and 'useless_folder' not in [parent.name for parent in pth.parents]]  It will work (probably - I didn't bother trying), but the problem is, useless_folder/ contains millions of files, and rglob will still traverse all of them, take ages, and only apply the filter when constructing the final list. Is there a way to tell Python not to waste time traversing useless folders (useless_folder/ in my case)?

Answer: You can easily write your own file iterator using recursion. def useless(path): # your logic to discard folders goes here ... def my_files_iter(path): if path.is_file(): yield path elif path.is_dir(): if useless(path): return for child_path in path.iterdir(): yield from my_files_iter(child_path)
Question: So, I want to use router within the websocket call. This is what I have, at the moment (using dummy data) version->v1->init.py router.include_router(websocket_controller, prefix="/ws") router_without_prefix.include_router(websocket_controller)  version->v1->websocket_controller.py So this is the enpoint: router = APIRouter() @router.websocket("/ws/something") async def websocket_endpoint(websocket: WebSocket):  but I want to remove this line: router_without_prefix.include_router(websocket_controller). But when I remove this line and call the websocket I have a 403. Can someone explain to me how I can correct this please?

Answer: Edit: According to this FastAPI GitHub issue, this issue is specific to websocket routers. Including the router using include_router might not work, but adding routes individually works. Like so: app.add_websocket_route(path="/ws/something", route=websocket_endpoint)  Old: Issue: The router has an extra prefix of /ws. You have a prefix of "/ws" defined in init.py, and the coroutine websocket_endpoint is accessed by "/ws/something". This means the async function is available on "/ws/ws/something". Removing the line, router_without_prefix.include_router(websocket_controller), means you also need to update the @router.websocket("/ws/something") line to just @router.websocket("/something"), since the router already has the prefix. Let me know if this works!
Question: How is the enumeration for QStyle.StandardPixmap defined in PyQt6? I've tried to replicate it as shown below: from enum import Enum class bootlegPixmap(Enum): SP_TitleBarMenuButton = 0 SP_TitleBarMinButton = 1 SP_TitleBarMaxButton = 2 for y in bootlegPixmap: print(y)  I get the following output: bootlegPixmap.SP_TitleBarMenuButton bootlegPixmap.SP_TitleBarMinButton bootlegPixmap.SP_TitleBarMaxButton  If I try and iterate over the original using the following code: from PyQt6.QtWidgets import QStyle for x in QStyle.StandardPixmap: print(x)  I get numerical values only: 0 1 2 ... 

Answer: This has got nothing to do with PyQt per se. It's just the normal behaviour of Python's enum classes. By default, the members of IntEnum/IntFlag print their numerical values; so if you want more informative output, you should use repr instead: >>> import enum >>> >>> class X(enum.IntEnum): A = 1 ... >>> f'{X.A} - {X.A!r}' 1 - <X.A: 1>  By contrast, the members of Enum have opaque values, so they default to a textual representation of the values: >>> class Y(enum.Enum): A = 1 ... >>> f'{Y.A} - {Y.A!r}' Y.A - <Y.A: 1>  Needless to say, PyQt always uses the integer classes, since that's closest to how they're defined in Qt: >>> from PyQt6.QtWidgets import QStyle >>> >>> QStyle.StandardPixmap.mro() [<enum 'StandardPixmap'>, <enum 'IntEnum'>, <class 'int'>, <enum 'ReprEnum'>, <enum 'Enum'>, <class 'object'>] >>> >>> f'{QStyle.StandardPixmap.SP_ArrowUp} - {QStyle.StandardPixmap.SP_ArrowUp!r}' >>> '50 - <StandardPixmap.SP_ArrowUp: 50>'
Question: So, I just stumbled upon a hurdle concerning the use of attrs, which is quite new to me (I guess this also applies to dataclasses?). I have two classes, one I want to use as an attribute for another. This is how I would do this with regular classes: class Address: def __init__(self) -> None: self.street = None class Person: def __init__(self, name) -> None: self.name = name self.address = Address()  Now with attrs, I tried to do the following: from attrs import define @define class Address: street: str | None = None @define class Person: name: str self.address = Address()  Now if I try the following, I don't get the same result for a class and a dataclass, for which the reason wasn't obvious to me at first: person_1 = Person("Joe") person_2 = Person("Jane") person_1.address.street = "street" person_2.address.street = "other_street" print(person_1.address.street)  I would expect the output to be "street", which is what happens with a regular class. But with attrs, the output is "other_street". I then compared the hashes of person_1.address and person_2.address, and voila, they are the same. After some thinking this is logical, with attrs I instantiate Address immediately, so everyone gets the same instance of Address, with regular classes I only instantiate them when I instantiate the parent class. Now, there is a fix available with attrs: from attrs import define, field @define class Address: street: str | None = None @define class Person: name: str address: Address = field(init=False) def __attrs_post_init__(self): self.address = Address()  But this seems really cumbersome to implement every time. Is there a nice solution to this? One way would be to put the instantiation of Address outside of the class like this: address_1 = Address() person_1 = Person("Joe", address)  But my issue with that is, that often I want to instantiate the class in an empty state (for example to seperate input from computed values), and this way adds an extra step to instantiation which I need to remember. So in conclusion: In this case, attrs, dataclass, pydantic etc. blur the line between what belongs to the class and what belongs to the instance, and in my case that led to an hour of "wtf happened here". So back to normal classes? I really like the default and validation possibilities of attrs though. Or is there a best practice way to handle this kind of setup?

Answer: You can use the factory argument to specify a callable that is called to return a new instance for the field during instantiation: from attrs import define, field @define class Address: street: str | None = None @define class Person: name: str address: Address = field(factory=Address)
Question: I am sometimes getting data that is either a string or int for 2 of my values. Im trying to figure out the best solution for handling this and I am using pydantic v2. Ive been studying the @field_validtaors(pre=True) but haven't been successful. Here's my code: class User(BaseModel): system_id: str db_id: str id: str record: dict isValid: bool @field_validator("id", mode='before') <<<--------- not working def transform_system_id_to_str(cls, value) -> str: return str(value) @field_validator("system_id", mode='before') <<<--------- not working def transform_system_id_to_str(cls, value: str | int) -> str: if isinstance(value, str): print('yes') return value return str(value) 

Answer: You can validate more than one fields in a field_validator. Try the following code: class User(BaseModel): system_id: str db_id: str id: str record: dict isValid: bool @field_validator("id","system_id", mode='before') def transform_id_to_str(cls, value) -> str: return str(value) u = User(system_id='199', db_id="1", id='99', record={}, isValid=True) print(u) ## system_id='199' db_id='1' id='99' record={} isValid=True u = User(system_id=199, db_id="1", id=99, record={}, isValid=True) print(u) ## system_id='199' db_id='1' id='99' record={} isValid=True   Before validators run before Pydantic's internal parsing and validation (e.g. coercion of a str to an int). https://docs.pydantic.dev/latest/concepts/validators/
Question: Suppose I want to implement common variable among objects using class variables (similar to static in java/c++). When I access class variable using object, it shows default value. Then I updaded class variable via object, its not updating class variable for other objects. It is showing old value in when class variable accessed via other objects or Class Name directly. Why is it so, and what is the way in python to make class/static variable (common among objects) # Class for Computer Science Student class CSStudent: stream = 'cse' # Class Variable def __init__(self,name): self.name = name # Instance Variable # Objects of CSStudent class a = CSStudent('Geek') b = CSStudent('Nerd') print(a.stream) # prints "cse" print(b.stream) # prints "cse" print(a.name) # prints "Geek" print(b.name) # prints "Nerd" # Class variables can be accessed using class # name also print(CSStudent.stream) # prints "cse" # Now if we change the stream for just a it won't be changed for b a.stream = 'ece' b.stream = 'abc' print(CSStudent.stream) # prints 'ece' print(a.stream) # prints 'ece' print(b.stream) # prints 'abc' 

Answer: You need to know the lookup procedure. First off, both instances and classes have their own namespaces. Only the class variables are shared between all instances. The namespace of classes and instances are accessible via __dict__ attribute. When you access an attribute from an instance, Python first looks at the namespace of the instance, if it can find it, it returns it, otherwise it's gonna find it in its class! class CSStudent: stream = "cse" a = CSStudent() b = CSStudent() print("stream" in a.__dict__) # False print("stream" in b.__dict__) # False print("stream" in CSStudent.__dict__) # True print(a.stream) # cse print(b.stream) # cse  So a and b doesn't have that stream, only the class has. Now a.stream = "something" will add that attribute to the namespace of that specific instance. a.stream = "something" print("stream" in a.__dict__) # True print("stream" in b.__dict__) # false print("stream" in CSStudent.__dict__) # True  Now if you access a.stream it finds it in the namespace of a and returns it, but because b doesn't have that, it will find it in the class.(the shared one) print(a.stream) # something print(b.stream) # cse  If you want to the change to reflect all instances, you need to change it on the class which has it shared between all instances. class CSStudent: stream = "cse" a = CSStudent() b = CSStudent() print(a.stream) # cse print(b.stream) # cse CSStudent.stream = "something" print(a.stream) # something print(b.stream) # something  Always think about who has what. Then consider the precedence, instance namespaces are checked first. Note: I simplified the explanation by not mentioning what descriptors are because we don't have one here. Take a look at it later. You would be surprised to see that in fact always class namespaces are checked first!
Question: Consider the following example: pd.DataFrame({'time' : [1,2,3], 'X1-Price': [10,12,11], 'X1-Quantity' : [2,3,4], 'X2-Price': [3,4,2], 'X2-Quantity' : [1,1,1]}) Out[92]: time X1-Price X1-Quantity X2-Price X2-Quantity 0 1 10 2 3 1 1 2 12 3 4 1 2 3 11 4 2 1  I am trying to reshape this wide dataframe into a long format. The difficulty is that I want to break down the variables by type (identified by the X1, X2 variables) while keeping price and quantity aligned on the same rows. That is, the desired output is the following Out[93]: time type quantity price 0 1 X1 2 10 1 2 X1 3 12 2 3 X1 4 11 3 1 X2 1 3 4 2 X2 1 4 5 3 X2 1 2  I am not sure how to do this. I tried the pd.wide_to_long() function but it does not work correctly. pd.wide_to_long(b, i = 'time', j = 'value', stubnames = ['X']) Out[95]: Empty DataFrame Columns: [X1-Price, X2-Price, X2-Quantity, X1-Quantity, X] Index: []  Do you have any idea? Thanks!

Answer: wide_to_long requires a specific order. First invert the chunks around the -, then change some of the default parameters: # alternative: # tmp.columns = df.columns.str.split('-').str[::-1].str.join('-') tmp = df.rename(columns=lambda s: '-'.join(s.split('-')[::-1])) out = (pd.wide_to_long(tmp, i='time', j='type', stubnames=['Quantity', 'Price'], sep='-', suffix=r'X\d+') .reset_index() )  Output:  time type Quantity Price 0 1 X1 2 10 1 2 X1 3 12 2 3 X1 4 11 3 1 X2 1 3 4 2 X2 1 4 5 3 X2 1 2
Question: I am reading a .fce file in Pandas. The start of the file looks like this: # Forces and moments acting on bodies # Direction1 = ( 1 0 0) # Direction2 = ( 0 1 0) # Moments around ( 0 0 0) # Boundary regions: 0 # Time F1-press F1-visc F1-total F2-press F2-visc F2-total M3-press M3-visc M3-total 1250 1.1630515 0.018437668 1.1814892 0.11872933 0.076623484 0.19535281 -0.079455905 -0.0024483534 -0.081904258 1250 1.1451982 0.019795173 1.1649934 0.10773641 0.076852597 0.18458901 -0.07645881 -0.0026731929 -0.079132003  I have been able to get pandas to ignore the first 5 lines since those are not useful to me. The headers are contained in the 6th line. The issue now is that Pandas is interpretting the # as one of the headers and I would like it to ignore this character. Time should be the first header. I am using the command df = pd.read_csv('DragLift.fce',sep='\s+', skiprows=5) to read the file. I have found that both read_table and read_csv work the same for this and have the same issue. Everything works fine except Pandas' interpretation of the header. The output is:     # Time F1-press ...     1250 1.163051 0.018438 ...   ... ... ... ...     when I want:     Time F1-press F1-visc ...     1250 1.163051 0.018438 ...   ... ... ... ...     How can I get Pandas to ignore the #? Thank you for your help.

Answer: Quick and easy option, shift your data after import: df = (pd.read_csv('DragLift.fce',sep='\s+', skiprows=5) .shift(axis=1).iloc[:, 1:] ) print(df)  Alternatively, read the headers with a regex, ignore the lines with #, and pass the names manually: import re with open('DragLift.fce') as f: m = next(re.finditer('#([^\n]+)\n(?!#)', f.read()), None) names = m.group(1).split() if m else None df = pd.read_csv('DragLift.fce', sep='\s+', names=names, comment='#')  Output:  Time F1-press F1-visc F1-total F2-press F2-visc F2-total M3-press M3-visc M3-total 0 1250 1.163051 0.018438 1.181489 0.118729 0.076623 0.195353 -0.079456 -0.002448 -0.081904 1 1250 1.145198 0.019795 1.164993 0.107736 0.076853 0.184589 -0.076459 -0.002673 -0.079132
Question: I am using Google Cloud Datastore via the python-ndb library (Python 3). My goal is to transactionally create two entities at once. For example, when a user creates an Account entity, also create a Profile entity, such that if either entity fails to be created, then neither entity should be created. From the datastore documentation, it can be implemented like this: from google.cloud import datastore client = datastore.Client() def transfer_funds(client, from_key, to_key, amount): with client.transaction(): from_account = client.get(from_key) to_account = client.get(to_key) from_account["balance"] -= amount to_account["balance"] += amount client.put_multi([from_account, to_account])  However, there is no example provided by the python-ndb documentation. From chatgpt, I tried something like this: from google.cloud import ndb client = ndb.Client() def create_new_account(): with client.context(): @ndb.transactional def _transaction(): account = Account() account.put() profile = Profile(parent=account.key) profile.put() return account, profile try: account, profile = _transaction() except Exception as e: print('Failed to create account with profile') raise e return account, profile  However, I get the error: TypeError: transactional_wrapper() missing 1 required positional argument: 'wrapped' 

Answer: You're almost there. It should be  @ndb.transactional() def _transaction():  Ran your code and got your error. Then amended it to mine and it went through. Source - https://github.com/googleapis/python-ndb/blob/main/google/cloud/ndb/_transaction.py#L317
Question: Edited to show more accurate strings I have a long list with the following grouping structure. I am showing just two groups, but in reality there are 300. The strings can be any random collection of numbers and letters: ["['P431', 'N7260', 'K492'], ['R109', 'T075X9A'], ['U8154', 'C861']", "['R878X8', 'T61'], ['F6332', 'Q7979'], ['L520'], ['B7939', 'K5132']"]  For each group in the list, I want the final output to look like the following: [['P431 N7260 K492 R109 T075X9A', 'U8154 C861'], ['R878X8 T61 F6332 Q7979 L520', 'B7939 K5132']]  So for each group in the list, I want to join all of the elements in every sub-list except the last sub-list, then separately join all of the elements in the last sub-list, and finally add these both together into a single sub-list, each of which will now have just two elements. Thank you in advance as I am a beginner.

Answer: You can try this: from ast import literal_eval example=[ "['P431', 'N7260', 'K492'], ['R109', 'T075X9A'], ['U8154', 'C861']", "['R878X8', 'T61'], ['F6332', 'Q7979'], ['L520'], ['B7939', 'K5132']", "['a','b','c'], 'not a list'" ] for st in example: *first_grp,last=literal_eval(st) if any(not isinstance(s, list) for s in (first_grp, last)): continue print([" ".join([e for r in first_grp for e in r]), " ".join(last)])  Prints: ['P431 N7260 K492 R109 T075X9A', 'U8154 C861'] ['R878X8 T61 F6332 Q7979 L520', 'B7939 K5132']  This is similar to Andrej Kesely solution but may overcome the 'random strings' you describe with a type check.