eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: Unable to concatenate the polars dataframes with data types of a column being f64 and i64. I have two pandas dataframes df1, df2 in pandas, where column 'a' in df1 is float and in df2 is int, when I perform pd.concat([df1, df2]) it works. However, when I try the same operation on polars dataframes, it is throwing the following error: exceptions.ShapeError: unable to vstack, dtypes for column "a" don't match: f64 and i64 pandas code: import pandas as pd df1 = pd.DataFrame({'a': [1.0, 2.0, 3.0], 'b': [1, 2, 3]}) df2 = pd.DataFrame({'a': [1, 2, 3], 'b': [1, 2, 3]})  pd.concat() produces the following output: pd.concat([pd_df1, pd_df2]) a b 0 1.00000000 1 1 2.00000000 2 2 3.00000000 3 0 1.00000000 1 1 2.00000000 2 2 3.00000000 3  polars code: import polars as pl df1 = pl.DataFrame({'a': [1.0, 2.0, 3.0], 'b': [1, 2, 3]}) df2 = pl.DataFrame({'a': [1, 2, 3], 'b': [1, 2, 3]})  pl.concat() is producing the error, unlike pandas. pl.concat([df1, df2]) Traceback (most recent call last): File "C:\Users\user\.conda\envs\dev\lib\site-packages\IPython\core\interactiveshell.py", line 3362, in run_code async def run_code(self, code_obj, result=None, *, async_=False): File "<ipython-input-16-4301449ba376>", line 1, in <cell line: 1> pl.concat([df1, df2]) File "C:\Users\user\.conda\envs\dev\lib\site-packages\polars\functions\eager.py", line 22, in concat def concat( exceptions.ShapeError: unable to vstack, dtypes for column "a" don't match: `f64` and `i64`  Here, I am fetching the data from database for various tables and creating a list of dataframes before concatenating them. Kindly help me with a solution where I could have the feasibility of not hard coding the column name in such scenarios.

Answer: You can use the vertical_relaxed strategy. pl.concat([df1, df2], how="vertical_relaxed")  shape: (6, 2) ┌─────┬─────┐ │ a ┆ b │ │ --- ┆ --- │ │ f64 ┆ i64 │ ╞═════╪═════╡ │ 1.0 ┆ 1 │ │ 2.0 ┆ 2 │ │ 3.0 ┆ 3 │ │ 1.0 ┆ 1 │ │ 2.0 ┆ 2 │ │ 3.0 ┆ 3 │ └─────┴─────┘
Question: I'm using the Ultralytics YOLOv8 implementation to perform object detection on an image. However, when I try to retrieve the classification probabilities using the probs attribute from the results object, it returns None. Here's my code: from ultralytics import YOLO # Load a model model = YOLO('yolov8n.pt') # pretrained YOLOv8n model # Run batched inference on a list of images results = model('00000.png') # return a list of Results objects # Process results list for result in results: boxes = result.boxes # Boxes object for bbox outputs masks = result.masks # Masks object for segmentation masks outputs keypoints = result.keypoints # Keypoints object for pose outputs probs = result.probs # Probs object for classification outputs print(probs)  When I run the above code, the output for print(probs) is None. The remaining output is image 1/1 00000.png: 640x640 1 person, 1 zebra, 7.8ms Speed: 2.6ms preprocess, 7.8ms inference, 1.3ms postprocess per image at shape (1, 3, 640, 640)  Why is the probs attribute returning None, and how can I retrieve the classification probabilities for each detected object? Is there a specific design reason behind this behavior in the Ultralytics YOLOv8 implementation?

Answer: I think you should be able to get the confidences with results[0].boxes.conf. The probs property seems not to be working in Yolov8
Question: I am trying to create a dictionary tree from child-parent relathionships but the issue here is that the child can contain one of his predecesors generating loops that I would like to avoid. list_child_parent = [(2,1),(3,2),(4,2),(5,3),(6,4),(2,6)] def make_map(list_child_parent): has_parent = set() all_items = {} for child, parent in list_child_parent: if parent not in all_items: all_items[parent] = {} if child not in all_items: all_items[child] = {} if parent not in all_items[child]: all_items[parent][child] = all_items[child] has_parent.add(child) else: continue result = {} for key, value in all_items.items(): if key not in has_parent: result[key] = value return result make_map(list_child_parent)  The code works well if a son contains his father but not if it is the granfather... I think another approach is needed. The expected results could be any if it makes sense: {1: {2: {3: {5: {}}, 4: {6: {}}}}} {1: {2: {3: {5: {}}, 4: {6: 2}}}} {1: {2: {3: {5: {}}, 4: {6: {2:{}}}}}} The current result is: {1: {2: {3: {5: {}}, 4: {6: {2: {...}}}}}}

Answer: You can use a set to keep track of all the children, so that you can make the dict of descendants an empty dict if the current child has previously been a child of another parent: def make_map(list_child_parent): mapping = {} children = set() for child, parent in list_child_parent: mapping.setdefault(parent, {})[child] = ( {} if child in children else mapping.setdefault(child, {}) ) children.add(child) return { parent: mapping[parent] for parent in mapping.keys() - children }  so that: make_map([(2, 1), (3, 2), (4, 2), (5, 3), (6, 4), (2, 6)])  returns: {1: {2: {3: {5: {}}, 4: {6: {2: {}}}}}}  Demo: Try it online!
Question: Is there a pure numpy way that I can use to get to this expected outcome? Right now I have to use Pandas and I would like to skip it. import pandas as pd import numpy as np listOfDicts = [{'key1': np.array(10), 'key2': np.array(10), 'key3': np.array(44)}, {'key1': np.array(2), 'key2': np.array(15), 'key3': np.array(22)}, {'key1': np.array(25), 'key2': np.array(25), 'key3': np.array(11)}, {'key1': np.array(35), 'key2': np.array(55), 'key3': np.array(22)}]  Use pandas to parse: # pandas can unpack simply df = pd.DataFrame(listOfDicts) # get all values under the same key xd = df.to_dict('list') # ultimate goal np.stack([v for k, v in xd.items() if k not in ['key1']], axis=1) array([[10, 44], [15, 22], [25, 11], [55, 22]])  # I would like listOfDicts to transform temporarily into this with pure numpy, # from which I could do basically anything to it: {'key1': [np.array([10, 2, 25, 35])], 'key2': [np.array([10, 15, 25, 55])], 'key3': [np.array([44, 22, 11, 22])] } 

Answer: One way to turn your dataframe into a dictionary of numpy arrays, is to transpose it and use DataFrame.agg() to merge the columns: import numpy as np import pandas as pd listOfDicts = [{'key1': np.array(10), 'key2': np.array(10), 'key3': np.array(44)}, {'key1': np.array(2), 'key2': np.array(15), 'key3': np.array(22)}, {'key1': np.array(25), 'key2': np.array(25), 'key3': np.array(11)}, {'key1': np.array(35), 'key2': np.array(55), 'key3': np.array(22)}] df.transpose().agg(np.stack, axis=1).to_dict() # {'key1': array([10, 2, 25, 35]), # 'key2': array([10, 15, 25, 55]), # 'key3': array([44, 22, 11, 22])}  If you just want the values, you can pull those out and stack them and then slice and dice with numpy: np.stack(df.transpose().agg(np.stack, axis=1).values) # array([[10, 2, 25, 35], # [10, 15, 25, 55], # [44, 22, 11, 22]])
Question: I stumbled upon this code that I found weird as it seems to violate the fact that python builtins call dunder methods directly from the class of the object. Using __call__ as an example, if we define class A as following: class A: @property def __call__(self): def inner(): return 'Called.' return inner a = A() a() # return 'Called.' type(a).__call__(a) # return 'property' object is not callable.  However, this behaviour seems to contradict with what's said in Python's official documentation:  object.__call__(self[, args...]) Called when the instance is “called” as a function; if this method is defined, x(arg1, arg2, ...) roughly translates to type(x).__call__(x, arg1, ...).  Can anyone explain what is going on here?

Answer: Yes, but it respects the way to retrieve a method from a given function - We can see that the __get__ method is called: On the code bellow, I just replaced property with a simpler descriptor that will retrieve its "func" - and used it as the __call__ method.  In [34]: class X: ...: def __init__(self, func): ...: self.func = func ...: def __get__(self, instance, owner): ...: print("descriptor getter called") ...: return self.func ...: In [35]: class Y: ...: __call__ = X(lambda: "Z") ...: In [36]: y = Y() In [37]: y() descriptor getter called Out[37]: 'Z'  So, the "dunder" functionality just retrieved the method through its __get__ as usual for all methods. What was skipped is the step that goes through __getattribute__ - Python will go directly to the __call__ slot in the A class, and not go through the normal lookup sequence, by calling __getattribute__, which starts at the class (for a descriptor), then looks at the instance, than back to the class (for a regular attribute): it assumes that if there is something at the instance's class dunder slot it is a method, and uses its __get__ method accordingly. A function's __get__ is used when retrieving it from an instance - as that is the mechanism that injects the instance as the self argument for the call. And __get__ is exactly the thing that property replaces to perform its "magic". To demonstrate that __getatribute__ is not called, in iPython, I had to encapsulate "y" inside a dummy function, otherwise iPython would trigger __getattribute__ when trying to autocomplete stuff:  In [42]: class Y: ...: __call__ = X(lambda: "Z") ...: def __getattribute__(self, name): ...: print("getattribute called") ...: return super().__getattribute__(name) ...: In [43]: def w(): ...: y = Y() ...: return y() ...: In [44]: w() descriptor getter called Out[44]: 'Z' # in contrast with: In [46]: Y().__call__() getattribute called descriptor getter called Out[46]: 'Z'
Question: Let's say I have a string like: 'abcdefgha'  I'd like to find the index of the next character a after the index 2 (in a circular manner). Meaning it should find index 7 in this case (via mystr.index('a', 2)); however, in this case: 'abcdefgh'  it should return index 0. Is there any such built-in function?

Answer: There isn't a builtin for this, but you can easily write a function: def index_circular(s: str, sub: str, n: int) -> int: try: # Search starting from n return s.index(sub, n) except ValueError: # Wrap around and search from the start, until n return s.index(sub, 0, n+len(sub)-1)  In use: >>> n = 2 >>> c = 'a' >>> index_circular('abcdefgha', c, n) 8 >>> index_circular('abcdefgh', c, n) 0 >>> index_circular('bcdefgh', c, n) Traceback (most recent call last): ... ValueError: substring not found  (Note that 'a' actually occurs at index 8, not 7, in the first case.) Note: In the second s.index call, I'm setting the end parameter in order to avoid searching parts of the string that have already been searched. This is a bit of a premature optimization, but it's also a bit of a clarification about exactly which parts of the string are being searched in each step. The +len(sub)-1 is to allow for multi-character sub that spans index n, like: >>> index_circular('abc', 'ab', 1) 0
Question: I'm encountering an issue with the following Python code where a ValueError is raised when trying to create an instance of an Enum with a value that doesn't correspond to any defined enum member: from enum import Enum class Option(Enum): OPTION_1 = "Option 1" OPTION_2 = "Option 2" NONE = "" def __new__(cls, value): try: obj = object.__new__(cls) obj._value_ = value return obj except ValueError: return Option.NONE tmp = Option("Option 3")  The ValueError seems to be related to how the __new__ method is handling the invalid values. I want the code to create an instance of the NONE member in case an invalid value is provided, but it doesn't seem to be working as expected. Why is this code resulting in a ValueError and how can I achieve the desired behavior of using the NONE member for invalid values?

Answer: The __new__ method is used during enum class creation; after that Enum.__new__ is swapped in, and it only does look-ups. To handle situations like these, use the _missing_ method: @classmethod def _missing_(cls, value): return cls.NONE   Disclosure: I am the author of the Python stdlib Enum, the enum34 backport, and the Advanced Enumeration (aenum) library.
Question: How can I create a regular expression in python that matches consecutive identical characters, regardless of whether line breaks or spaces are in between? The number of identical characters should be adjustable. examples (e can be any character except newline or space): match: eee, e e e, e e e no match: ebe, e b e, e e attempts: (\S)[\s\n]*\1{2} (\S)(?:\s|\n)*\1{2} 

Answer: You may use this regex: \A\s*(\S)(?:\s*\1\s*)+\Z  RegEx Demo RegEx Details:  \A: asserts position at start of the string \s*: Match 0 or more whitespaces (\S): Match any non-whitespace character and capture in group #1 (?:\s*\1\s*)+: Match same value we captured in group #1 surrounded with 0 or more whitespaces on either side. Repeat this group 1 or more times \Z: asserts position at the end of the string
Question: I'm learning python so I'm not an expert. I have a string like the following one (the line break is \n): test1 test2 othertext test3 test4  I would like to remove the othertext line but I can't figure out how to do it. I want this: test1 test2 test3 test4  Othertext is a text written by hand so I have to consider the possible errors: "OTherTEXT", "other text", "Othe rtext". It's always in one line, so maybe a .strip() could be enough? Any idea on how can I make this easily? Should I make a loop that "strips" every part of text between \n and the following \n? I tried to remove the text with a .replace("othertext", "") but it's a weak implementation.

Answer: Case insensitivity doesn't ignore spaces. So you have to first remove the spaces yourself (with .replace(" ", "") then do the comparison. For comparison part, use .lower() to convert the text into its lowercase version. text = """test1 test2 otHeR teXT test3 test4""" def remove_line(text: str, word: str) -> str: return "\n".join( [line for line in text.splitlines() if line.replace(" ", "").lower() != word] ) print(remove_line(text, "othertext"))  output: test1 test2 test3 test4
Question: I have a global configuration pp that changes at runtime and needs it to be shared across all parent/child objects. class Config: pp = 'Init' def __init__(self): pass class Child(Config): def __init__(self, name): self.cc = name par = Config() print(f"Parent: {par.pp}") par.pp = "123" print(f"Parent: {par.pp}") child = Child('XYZ') print(f"Child-1: {child.pp} - {child.cc}")  This prints: Parent: Init Parent: 123 Child-1: Init - XYZ  The third line is expected to be Child-1: 123 - XYZ How can I implement that in a clean way? UPDATE* Currently it works with a method like: class Config: pp = 'Init' def __init__(self): pass def set_pp(self, val): type(self).pp = val 

Answer: In your example, the parent of Child is not par (which is an instance of Config, but it is the class Config. So you could change the value directly on the class, like that: class Config: pp = 'Init' def __init__(self): pass class Child(Config): def __init__(self, name): self.cc = name print(f"Parent: {Config.pp}") Config.pp = "123" print(f"Parent: {Config.pp}") child = Child('XYZ') print(f"Child-1: {child.pp} - {child.cc}")  Note that if you want to keep the same syntax, you could write class Config: _pp = "Init" @property def pp(self): return self._pp @pp.setter def pp(self, value): self.__class__._pp = value def __init__(self): pass class Child(Config): def __init__(self, name): self.cc = name par = Config() print(f"Parent: {par.pp}") par.pp = "123" print(f"Parent: {par.pp}") child = Child('XYZ') print(f"Child-1: {child.pp} - {child.cc}")  In this version, the pp is set from the parent object, to the class Config.
Question: I am currently developing a Python script to retrieve a comprehensive list of all the jobs that were executed yesterday. However, I'm encountering an issue with the script's pagination mechanism using tokens. Despite my attempts to loop through the pagination process, the resulting output remains unchanged. Here is the code import requests import pandas as pd import math import datetime import json def fetch_and_process_job_runs(base_uri, api_token, params): endpoint = '/api/2.1/jobs/runs/list' headers = {'Authorization': f'Bearer {api_token}'} all_data = [] # To store all the data from multiple pages while True: # print(params) response = requests.get(base_uri + endpoint, headers=headers, params=params) response_json = response.json() data = [] for run in response_json["runs"]: start_time_ms = run["start_time"] start_time_seconds = start_time_ms / 1000 start_time_readable = datetime.datetime.fromtimestamp(start_time_seconds).strftime('%Y-%m-%d %H:%M:%S') data.append({ "job_id": run["job_id"], "creator_user_name": run["creator_user_name"], "run_name": run["run_name"], "run_page_url": run["run_page_url"], "run_id": run["run_id"], "execution_duration_in_mins": math.ceil(int(run.get('execution_duration')) / (1000 * 60)), "result_state": run["state"].get("result_state"), "start_time": start_time_readable }) all_data.extend(data) df = pd.DataFrame(all_data) print(df) if response_json.get("has_more") == True: next_page_token = response_json.get("next_page_token") params['next_page_token'] = next_page_token else: break df = pd.DataFrame(all_data) return df # Replace with your actual values now = datetime.datetime.utcnow() yesterday = now - datetime.timedelta(days=1) start_time_from = int(yesterday.replace(hour=0, minute=0, second=0, microsecond=0).timestamp()) * 1000 start_time_to = int(yesterday.replace(hour=23, minute=59, second=59, microsecond=999999).timestamp()) * 1000 params = { # "start_time_from": start_time_from, # "start_time_to": start_time_to, "expand_tasks": True } baseURI = 'https://adb-xxxxxxxxxxxxxx.azuredatabricks.net' apiToken = 'xxxxxxxxxxxxxxxxxxxxxxxxxx' result_df = fetch_and_process_job_runs(baseURI, apiToken, params) print(result_df)  Please help me.

Answer: I noticed that the value of next_token wasn't changing in API response and then figured that you have a very small error in your code. The parameter to be passed in request is page_token and not next_page_token. As per documentation at https://docs.databricks.com/api/workspace/jobs/list, page_token string Use next_page_token or prev_page_token returned from the previous request to list the next or previous page of jobs respectively. So params['next_page_token'] needs to change to params['page_token']
Question: I have multiple columns that contained only 0s or 1s.     Apple Orange Pear     1 0 1   0 0 1   1 1 0     I would like to count and input the number of 0s (in "Wrong" column) and 1s (in "Correct" column) of each column in the new dataframe, and total them up into a table that looks like the following.     Fruit Correct Wrong     Apple 2 1   Orange 1 2   Pear 2 1     I tried a blend of value_counts(), groupby(), and pandas.pivot_table, but got stuck with the manipulation of the table.

Answer: Try this: df.apply(pd.Series.value_counts).rename(index={0:'Wrong', 1:'Correct'}).T  Use pd.DataFrame.apply to "apply" pd.Series.value_counts to each column of the dataframe, then rename the index values using a dictionary for 0 and 1 to Wrong and Correct. Lastly, use T to transpose the dataframe. Output:  Wrong Correct Apple 1 2 Orange 2 1 Pear 1 2  And, you can add .rename_axis('Fruit').reset_index() to get:  Fruit Wrong Correct 0 Apple 1 2 1 Orange 2 1 2 Pear 1 2
Question: I am using this cmd on windows to upload my package in testpypi twine upload -r testpypi dist/* but it's showing this error. so how to upload in testpypi and pypi when 2FA is enable ? Uploading mypackage-0.1.0-py3-none-any.whl 100% ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 8.5/8.5 kB • 00:00 • ? WARNING Error during upload. Retry with the --verbose option for more details. ERROR HTTPError: 401 Unauthorized from https://test.pypi.org/legacy/ User has two factor auth enabled, an API Token or Trusted Publisher must be used to upload in place of password. 

Answer: create API token from your pypi account then give username =__token__ and password=APITokenafter cmd twine upload -r testpypi dist/* or if you are using Twine to upload your projects to PyPI, set up your $HOME/.pypirc file like this: [testpypi] username = __token__ password = API Token
Question: I want pathlib.Path to automatically output logs for some destructive commands such as path.rename(new_path). I made a subclass of pathlib.Path with logging functions, and replaced from pathlib import Path to from mylib import MyPath as Path. But it does not affect to the existing subclasses of pathlib.Path such as pathlib.WindowsPath, which is the actual implementation class of path instances. from pathlib import Path from mynicelib import MyPath p = MyPath('/path/to/file') isinstance(p, MyPath) # -> False isinstance(p, Path) # -> True type(p) # -> <class 'pathlib.WindowsPath'> 

Answer: Just do some monkeypatching: from pathlib import Path Path.oldrename = Path.rename def rename(self,b): print("Inside my rename") self.oldrename(b) Path.rename = rename p = Path('./x.c') p.rename('y.c')
Question: I have a dataframe which represents features of a linear regression model. df1 = pd.DataFrame({'yyyyww': ['2022-01','2022-02','2022-03', '2022-04','2022-05','2022-06','2022-07','2022-08','2022-09','2022-10'], 'feature1': [1000,2000,4000,3000,5000,2000,8000,2000,4000,3000], 'feature2': [9000,7000,3000,1000,2000,3000,6000,8000,1000,1000], 'feature3': [3000,1000,2000,5000,9000,7000,2000,3000,5000,9000]})   I run the model and calculate the coefficients which produces another dataframe, below. df2 = pd.DataFrame({'feature': ['feature1','feature2','feature3'], 'coefficient': [-1,2,0.5]})   I then want to produce a third dataframe where the contents are the product of the values from df1 and the corresponding coefficients from df2. Desired output below. df3 = pd.DataFrame({'yyyyww': ['2022-01','2022-02','2022-03', '2022-04','2022-05','2022-06','2022-07','2022-08','2022-09','2022-10'], 'feature1': [-1000,-2000,-4000,-3000,-5000,-2000,-8000,-2000,-4000,-3000], 'feature2': [18000,14000,6000,2000,4000,6000,12000,16000,2000,2000], 'feature3': [1500,500,1000,2500,4500,3500,1000,1500,2500,4500]})   I have tried to achieve this using mul and multiply in the following manner, however this does not produce the desired result. features = [['feature1', 'feature2', 'feature3']] results = pd.DataFrame() for cols in features: results[cols] = df1[cols] results = df1.mul(df2['coefficient'], axis =0) results 

Answer: Try this using pandas intrinsic data alignment tenet: df1.set_index('yyyyww').mul(df2.set_index('feature')['coefficient'])  Output:  feature1 feature2 feature3 yyyyww 2022-01 -1000.0 18000.0 1500.0 2022-02 -2000.0 14000.0 500.0 2022-03 -4000.0 6000.0 1000.0 2022-04 -3000.0 2000.0 2500.0 2022-05 -5000.0 4000.0 4500.0 2022-06 -2000.0 6000.0 3500.0 2022-07 -8000.0 12000.0 1000.0 2022-08 -2000.0 16000.0 1500.0 2022-09 -4000.0 2000.0 2500.0 2022-10 -3000.0 2000.0 4500.0