eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I'm struggling to create a new list based on two input lists. Here's an example: data_1 = [ { "title": "System", "priority": "medium", "subtitle": "mason", "files": [ {"name": "mason", "path": "/tmp/mason/mason.json"}, {"name": "mason", "path": "/tmp/mason/build.json"} ]}, { "title": "System", "priority": "medium", "subtitle": "kylie", "files": [ {"name": "kylie", "path": "/tmp/kylie/build.tar"}, {"name": "kylie", "path": "/tmp/kylie/kylie.json"} ]} ] data_2 = [ { "title": "System", "priority": "medium", "files": [ {"name": "build", "path": "/tmp/kylie/build.tar"}, {"name": "kylie", "path": "/tmp/kylie/kylie.json"}, {"name": "mason", "path": "/tmp/mason/mason.json"}, {"name": "build", "path": "/tmp/mason/build.json"} ]} ] merged = [ { "title": "System", "priority": "medium", "subtitle": "mason", "files": [ {"name": "mason", "path": "/tmp/mason/mason.json"}, {"name": "build", "path": "/tmp/mason/build.json"} ]}, { "title": "System", "priority": "medium", "subtitle": "kylie", "files": [ {"name": "build", "path": "/tmp/kylie/build.tar"}, {"name": "kylie", "path": "/tmp/kylie/kylie.json"} ]} ]  Basically, it would be OK to keep data from data_1 list and just replace files.name with the names coming from data_2 list, having the same path. Or multiply the element in data_2, assigning a subtitle and deleting all data_2.files items that don't match the paths specified in data_1 list. Anyway, I started doing something like: for d1 in data_1: for d2 in data_2: d1_key = (d1.get("title"), d1.get("priority")) d2_key = (d1.get("title"), d1.get("priority")) if d1_key == d2_key: for df in d1.get("files"): if any(df.get("path") in entry.get("path") for entry in d2.get("files")): print("Help! There must be easier way!")  And I don't think it's clear enough to get into this further. Would you suggest any other way to get the merged list created?

Answer: You can create a reverse mapping that maps paths in data_2 to names, and iteratively modify all names in data_1 with the paths mapped with the mapping: from itertools import chain from operator import itemgetter name_of = dict( map( itemgetter('path', 'name'), chain.from_iterable(map(itemgetter('files'), data_2)) ) ) for record in data_1: for file in record['files']: file['name'] = name_of[file['path']]  data_1 becomes: [{'title': 'System', 'priority': 'medium', 'subtitle': 'mason', 'files': [{'name': 'mason', 'path': '/tmp/mason/mason.json'}, {'name': 'build', 'path': '/tmp/mason/build.json'}]}, {'title': 'System', 'priority': 'medium', 'subtitle': 'kylie', 'files': [{'name': 'build', 'path': '/tmp/kylie/build.tar'}, {'name': 'kylie', 'path': '/tmp/kylie/kylie.json'}]}]  You can deepcopy data_1 to a new list named merged first if you prefer not to modify data_1. Demo here
Question: I have a camera (EO Edmund Optics Camera, Model UI-154xSE-M) connected to a Windows computer via USB. When I open IDS Camera Manager, The camera is "configured correctly and can be opened". I am trying to write a code in Python to have the camera capture an image every x minutes for a period of y total minutes using the cv2 package. Unfortunately, I am getting an error while doing this. Please find my code below and the result: import cv2 import os import time from datetime import datetime, timedelta output_directory = camera = cv2.VideoCapture(0) if not camera.isOpened(): print("Error: Could not access the camera.") else: print("Camera found and initialized. Starting photo capture...") end_time = datetime.now() + timedelta(hours=24) capture_interval = 10 * 60 # Capture every 10 minutes (600 seconds) try: while datetime.now() < end_time: ret, frame = camera.read() print(f"Capture status: {ret}") # Print whether the capture was successful if not ret: print("Error: Failed to capture image.") continue # Skip this iteration and try again timestamp = datetime.now().strftime("%Y%m%d_%H%M%S") file_path = os.path.join(output_directory, f"photo_{timestamp}.jpg") cv2.imwrite(file_path, frame) print(f"Photo captured and saved at {file_path}") time.sleep(capture_interval) except KeyboardInterrupt: print("Photo capture interrupted by user.") finally: camera.release() print("Camera released. Photo capture ended.")  "Error: Failed to capture image." prints for the time frame I specify. Although "Camera found and initialized. Starting photo capture..." prints as well. I am very unfamiliar with accessing/controlling external devices with Python code. I am open to alternate ways to do this besides Python. I am new to coding and devices.

Answer: So isOpened() says True, but the read() call returns False? It might be worth trying the read call several times, instead of failing at the first sign of trouble. Some cameras are weird like that. OpenCV also has this quirk where it tries to set 640 x 480 resolution. If a camera doesn't support that, the interaction will fail. Try setting a supported resolution, e.g. the sensor's preferred video mode resolution (1280 x 1024 was it?)  OpenCV struggles because that camera doesn't appear to support normal camera APIs, such as USB Unified Video Class (UVC). That is the interface supported by all webcams and most modern frame grabbers and capture cards. Your camera, being an industrial camera, probably supports either "GigE Vision" or "USB3 Vision", which are standards for industrial cameras. OpenCV might support that, if you compile it to do that. To my knowledge, no (official or third-party) binary build of OpenCV for Python has such support. The manufacturer's own site links to a Python library you can use to access your camera: https://pypi.org/project/pyueye  I just checked the spec sheet for the part number you gave. The model has been discontinued. You might be out of luck in finding any support for it. The spec sheet doesn't speak about USB 3, so it's got to be USB 2, and that never, to my knowledge, carried any data conforming to an industrial vision API.
Question: In python, consider the following example from enum import StrEnum from typing import Literal, overload class A(StrEnum): X = "X" Y = "Y" class X: ... class Y: ... @overload def enum_to_cls(var: Literal[A.X]) -> type[X]: ... @overload def enum_to_cls(var: Literal[A.Y]) -> type[Y]: ... def enum_to_cls(var: A) -> type[X] | type[Y]: match var: case A.X: return X case A.Y: return Y case _: raise ValueError(f"Unknown enum value: {var}")  When I attempt to call enum_to_cls, I get a type error, with the following case: selected_enum = random.choice([x for x in A]) enum_to_cls(selected_enum) # Argument of type "A" cannot be assigned to parameter "var" of type "Literal[A.Y]" in # function "enum_to_cls" # "A" is not assignable to type "Literal[A.Y]" [reportArgumentType]  I understand the error and it makes sense, but I wanted to know, if there is any way to avoid this error. I know I can avoid this error, creating a branch for each enum case but then I am back to square one of why I wanted to created the function enum_to_cls.

Answer: The simple workaround is to include the implementation's signature as a third overload: (playgrounds: Pyright, Mypy) @overload def enum_to_cls(var: Literal[A.X]) -> type[X]: ... @overload def enum_to_cls(var: Literal[A.Y]) -> type[Y]: ... @overload def enum_to_cls(var: A) -> type[X] | type[Y]: ... def enum_to_cls(var: A) -> type[X] | type[Y]: # Implementation goes here  selected_enum = random.choice([x for x in A]) reveal_type(enum_to_cls(selected_enum)) # type[X] | type[Y]  This is a workaround rather than a definitive solution, because the enum A is supposed to be expanded to the union of its members during overload evaluation. Indeed, if the argument were declared to be of the type Literal[A.X, A.Y], there would be no error: (playgrounds: Pyright, Mypy) @overload def enum_to_cls(var: Literal[A.X]) -> type[X]: ... @overload def enum_to_cls(var: Literal[A.Y]) -> type[Y]: ... def enum_to_cls(var: A) -> type[X] | type[Y]: # Implementation goes here  selected_enum: Literal[A.X, A.Y] = ... reveal_type(enum_to_cls(selected_enum)) # type[X] | type[Y]  Type expansion during overload evaluation is part of a recent proposed addition to the specification. Pyright has yet to conform to this, but it will once the proposal is accepted.
Question: I have a problem where I need to access a MS SQL DB, so naturally it's naming convention is different: TableName(Id, ColumnName1, ColumnName2, ABBREVIATED,...) The way I got the model constructed in Python: class TableName(Base): __tablename__ = 'TableName' id = Column('Id', BigInteger, primary_key=True, autoincrement=True) order_id = Column('OrderId', BigInteger, ForeignKey('Orders.Id'), nullable=False) column_name1 = Column('ColumnName1', String(50), nullable=True) column_name2 = Column('ColumnName2', String(50), nullable=True) abbreviated= Column('ABBREVIATED', String(50), nullable=False) ...  Then, I have a Repository: class TableNameRepository: def __init__(self, connection_string: str): self.engine = create_engine(connection_string) self.Session = sessionmaker(bind=self.engine) def get_entry(self, order_id: int, column_name1: str) -> Optional[TableName]: with self.Session() as session: return session.query(TableName).filter_by( order_id=order_id, column_name1=column_name1 ).first()  This works, however, now to my actual problem: I have a template text file with placeholders using the SQL column names, meaning there are Id, ColumnName1, ColumnName2, ABBREVIATED strings and I need to replace them with values from the model. Here is how I tried to approach this: retrieved_table_entity = self.repository.get_entry(order_id, something) attr_to_column = { column.key: column.name for column in retrieved_table_entity.__table__.columns }  The problem is that doesnt work: It still creates keys that look like: {'Id': 'Id', 'OrderId': 'OrderId', 'ColumnName1': 'ColumnName1', ...} And so, the next step which is:  for attr_name, column_name in attr_to_column.items(): value = getattr(retrieved_table_entity, attr_name, None) logging.debug(f"Accessing {attr_name}: {value}") reg_dict[column_name] = '' if value is None else str(value)  Doesnt work because the keys are not the python keys, in debug mode testing, if I manually do like: getattr(retrieved_table_entity, 'id'), it returns the value. To summarize: I need a way to map, so that I have key/value dictionary like: { Id: 1 OrderID: 123 ColumnName1: 'SomeValue' }  And then I could simply to soemthing like self.template.format(**keys_to_values_dict) I have tried consulting with LLMs, and they keep circling around not suggesting a way to do this, I dont want to manually map the model again in this service function. I also reviewed the Mapping API and cant find something relevant: https://docs.sqlalchemy.org/en/14/orm/mapping_api.html For reference, my current dependencies: sqlalchemy==2.0.36 pymssql==2.2.11 

Answer: Untested but something like this: from sqlalchemy import inspect old_to_new = {c.name: k for k, c in inspect(TableName).columns.items()}  https://docs.sqlalchemy.org/en/20/orm/mapping_api.html#sqlalchemy.orm.Mapper.columns https://docs.sqlalchemy.org/en/20/orm/mapping_styles.html#inspection-of-mapper-objects
Question: I need to correctly outline as many as possible of the mortared stones in a street zone. The code below correctly detects some of them in the stones image "in.jpg", but it is not obvious why many remain undetected or only partly outlined. I'd also like to fix cases like contour 56, 57 or 92 by exploiting the fact that the stones are aligned and mostly oval image with detected contours "out.jpg". I know how to remove the obvious outliers, but before that I need to improve the detection method. import cv2 img = cv2.imread('in.jpg') gray = cv2.bitwise_not(img) gray = cv2.cvtColor(gray, cv2.COLOR_BGR2GRAY) gray = cv2.GaussianBlur(gray, (113, 113), 0) thresh = cv2.adaptiveThreshold(gray, 255, cv2.ADAPTIVE_THRESH_GAUSSIAN_C, cv2.THRESH_BINARY_INV, 61, 2) thresh = cv2.dilate(thresh, None, iterations=5) thresh = cv2.erode(thresh, None, iterations=1) cnts, _ = cv2.findContours(thresh, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE) print(f'Found {len(cnts)} contours') cv2.drawContours(img, cnts, -1, (0, 255, 0), 2) centroids = [cv2.moments(c) for c in cnts] for i, c in enumerate(centroids): cv2.putText(img, str(i), (int(c['m10'] / c['m00']), int(c['m01'] / c['m00']) - 5), cv2.FONT_HERSHEY_SIMPLEX, 0.6, (0, 0, 255), 2) cv2.imwrite('out.jpg', img)  The various parameters (blur 113, etc.) have been obtained by trial and error. It is not enough to just count the stones, I need to get their shape as closely as possible. Thank you for your suggestions! Contributing to Markus' question: here is an example of a mixed zone: 

Answer: You can use cv2.moments(contour) for calculating the ratio between the contour length and the contour area. This will let you rule out non oval contours. Having said that, this is a relatively difficult problem for a classical CV approach. A neural network will do a better job (given enough training data) than findContours. This is mainly because the stones are quite different in their appearance and modeling them using a classical model requires many parameters that will capture this variance. Neural networks are naturally better suited for that kind of task. There are off the shelf segmenters like the SAM model that can do the job and a quick search shows there are also models designed specifically for stones segmentation. Here is an example of using SAM2 segmentation:  You can find a colab here
Question: I have a polars dataframe df which has a datetime column date. I'm trying to get the name of the day and month of that column. Consider the following example. import polars as pl from datetime import datetime df = pl.DataFrame({ "date": [datetime(2024, 10, 1), datetime(2024, 11, 2)] })  I was hoping that I could pass parameter to month() or weekday() to get the desired format. df.with_columns( pl.col("date").dt.month().alias("Month"), pl.col("date").dt.weekday().alias("Day") )  shape: (2, 3) ┌─────────────────────┬───────┬─────┐ │ date ┆ Month ┆ Day │ │ --- ┆ --- ┆ --- │ │ datetime[μs] ┆ i8 ┆ i8 │ ╞═════════════════════╪═══════╪═════╡ │ 2024-10-01 00:00:00 ┆ 10 ┆ 2 │ │ 2024-11-02 00:00:00 ┆ 11 ┆ 6 │ └─────────────────────┴───────┴─────┘  However, this does not seem to be the case. My desired output looks as follows. shape: (2, 3) ┌─────────────────────┬───────┬──────────┐ │ date ┆ Month ┆ Day │ │ --- ┆ --- ┆ --- │ │ datetime[μs] ┆ str ┆ str │ ╞═════════════════════╪═══════╪══════════╡ │ 2024-10-01 00:00:00 ┆ Oct ┆ Tuesday │ │ 2024-11-02 00:00:00 ┆ Nov ┆ Saturday │ └─────────────────────┴───────┴──────────┘  How can I extract the day and month name from the date column?

Answer: You can use pl.Expr.dt.strftime to convert a date / time / datetime column into a string column of a given format. The format can be specified using the chrono strftime format. In your specific example, the following specifiers might be of interest:  %B for the full month name. %b for the abbreviated month name. Always 3 letters. %A for the full weekday name. %a for the abbreviated weekday name. Always 3 letters.  df.with_columns( pl.col("date").dt.strftime("%B").alias("month"), pl.col("date").dt.strftime("%b").alias("month (short)"), pl.col("date").dt.strftime("%A").alias("day"), pl.col("date").dt.strftime("%a").alias("day (short)"), )  shape: (2, 5) ┌─────────────────────┬──────────┬───────────────┬──────────┬─────────────┐ │ date ┆ month ┆ month (short) ┆ day ┆ day (short) │ │ --- ┆ --- ┆ --- ┆ --- ┆ --- │ │ datetime[μs] ┆ str ┆ str ┆ str ┆ str │ ╞═════════════════════╪══════════╪═══════════════╪══════════╪═════════════╡ │ 2024-10-01 00:00:00 ┆ October ┆ Oct ┆ Tuesday ┆ Tue │ │ 2024-11-02 00:00:00 ┆ November ┆ Nov ┆ Saturday ┆ Sat │ └─────────────────────┴──────────┴───────────────┴──────────┴─────────────┘
Question: I have some data that is saved in a dictionary of dataframes. The real data is much bigger with index up to 3000 and more columns. In the end I want to make a violinplot of two of the columns in the dataframes but for multiple dictionary entries. The dictionary has a tuple as a key and I want to gather all entries which first number is the same.  import pandas as pd import numpy as np import matplotlib.pyplot as plt data_dict = { (5, 1): pd.DataFrame({"Data_1": [0.235954, 0.739301, 0.443639], "Data_2": [0.069884, 0.236283, 0.458250], "Data_3": [0.170902, 0.496346, 0.399278], "Data_4": [0.888658, 0.591893, 0.381895]}), (5, 2): pd.DataFrame({"Data_1": [0.806812, 0.224321, 0.504660], "Data_2": [0.070355, 0.943047, 0.579285], "Data_3": [0.526866, 0.251339, 0.600688], "Data_4": [0.283107, 0.409486, 0.307315]}), (7, 3): pd.DataFrame({"Data_1": [0.415159, 0.834547, 0.170972], "Data_2": [0.125926, 0.401789, 0.759203], "Data_3": [0.398494, 0.587857, 0.130558], "Data_4": [0.202393, 0.395692, 0.035602]}), (7, 4): pd.DataFrame({"Data_1": [0.923432, 0.622174, 0.185039], "Data_2": [0.759154, 0.126699, 0.783596], "Data_3": [0.075643, 0.287721, 0.939428], "Data_4": [0.983739, 0.738550, 0.108639]}) }  My idea was that I could re-arrange it into a different dictionary and then plot the violinplot. Say that 'Data_1' and 'Data_4' are of interest. So then I loop over the keys in dict as below. new_dict = {} for col in ['Data_1','Data_4']: df = pd.DataFrame() for i in [5,7]: temp = [] for key, value in dict.items(): if key[0]==i: temp.extend(value[col]) df[i] = temp new_dict[col] = df  This then make the following dict. new_dict = {'Data_1': 5 7 0 0.235954 0.415159 1 0.739301 0.834547 2 0.443639 0.170972 3 0.806812 0.923432 4 0.224321 0.622174 5 0.504660 0.185039, 'Data_4': 5 7 0 0.888658 0.202393 1 0.591893 0.395692 2 0.381895 0.035602 3 0.283107 0.983739 4 0.409486 0.738550 5 0.307315 0.108639}  Which I then loop over to make the violin plots for Data_1and Data_4. for key, value in new_dict.items(): fig, ax = plt.subplots() ax.violinplot(value, showmeans= True) ax.set(title = key, xlabel = 'Section', ylabel = 'Value') ax.set_xticks(np.arange(1,3), labels=['5','7'])  While I get the desired result it's very cumbersome to re-arrange the dictionary. Could this be done in a faster way? Since it's the same column I want for each dictionary entry I feel that it should.

Answer: You could minimize the reshaping by using concat+melt and a higher level plotting library like seaborn: import seaborn as sns sns.catplot(data=pd.concat(data_dict, names=['section', None]) [['Data_1', 'Data_4']] .melt(ignore_index=False, var_name='dataset') .reset_index(), row='dataset', x='section', y='value', kind='violin', )  Output:  Another approach to reshape: tmp = (pd .concat(data_dict, names=['section', None]) [['Data_1', 'Data_4']] .pipe(lambda x: x.set_axis(pd.MultiIndex.from_arrays([x.index.get_level_values('section'), x.groupby('section').cumcount()]))) .T.stack() ) # then access the datasets tmp.loc['Data_1'] # section 5 7 # 0 0.235954 0.415159 # 1 0.739301 0.834547 # 2 0.443639 0.170972 # 3 0.806812 0.923432 # 4 0.224321 0.622174 # 5 0.504660 0.185039
Question: I have a DataFrame with a binary column that represents the hexadecimal encoding of an initial string:    random_id random_id_cesu8     123456789012 [31 32 33 34 35 36 37 38 39 30 31 32]    The random_id_cesu8 column contains the binary representation of the random_id string encoded in UTF-8 and displayed as a list of byte values in hexadecimal format. I want to transform the random_id_cesu8 column into a single concatenated hexadecimal string: 313233343536373839303132, which is the concatenation of each individual byte value from the random_id_cesu8 list. I’ve tried multiple approaches, but they all result in the original random_id value (123456789012) instead of the desired concatenated hexadecimal string (313233343536373839303132). How can I correctly achieve this transformation?

Answer: Here's a solution using PySpark to convert the binary column to its hexadecimal representation: from pyspark.sql import functions as F # Create DataFrame with the initial value df = spark.createDataFrame([("123456789012",)], ["value"]) # Convert the value to UTF-8 encoding and then to hex df = df.withColumn( "encoded_hex", F.hex(F.encode(F.col("value"), 'utf-8')), ) # Display the resulting DataFrame df.show(truncate=False)  Explanation:  encode(df.value, 'utf-8'): This function encodes the string value into a binary format using UTF-8 encoding. hex(): This function converts the binary data into its hexadecimal representation.  The result will be a DataFrame with the encoded_hex column containing the desired hexadecimal string 313233343536373839303132. +------------+------------------------+ |value |encoded_hex | +------------+------------------------+ |123456789012|313233343536373839303132| +------------+------------------------+
Question: Hi I want to define a polars schema. It works fine without a datetime format. However it fails with pl.Datetime. import polars as pl testing_schema: pl.Schema = pl.Schema( { "date": pl.Datetime, "some_int": pl.Int64, "some_str": pl.Utf8, "some_cost": pl.Float64, }, )  The error: lib/python3.11/site-packages/polars/schema.py", line 47, in _check_dtype raise TypeError(msg) TypeError: dtypes must be fully-specified, got: Datetime  Unfortunately I do not have any idea on how to fix it.

Answer: If you look at pl.Datetime, you'll find that it is initialized with parameters: class polars.datatypes.Datetime( time_unit: TimeUnit = 'us', time_zone: str | timezone | None = None )  Unlike, say, pl.Int64. Hence, you need to add the parentheses at the end, (), to use the defaults, or to pass arguments and override them. import polars as pl testing_schema: pl.Schema = pl.Schema( { "date": pl.Datetime(), "date2": pl.Datetime(time_unit="ns", time_zone="UTC"), "some_int": pl.Int64, }, )
Question: I know it's inadvisable to use eval() on untrusted input, but I want to see where this sanitiser fails. It uses a whitelist to only allow harmless builtins, and it immediately bails if there are any dunder properties called. (Note: the reason it string searches .__ and not just __ is because I want to allow things like foo.bar__baz). def safe_eval(code: str) -> Any | None: if '.__' in code: raise ValueError allowed_builtins = [ 'abs', 'all', 'any', 'ascii', 'bin', 'bool', 'bytearray', 'bytes', 'callable', 'chr', 'complex', 'dict', 'divmod', 'enumerate', 'filter', 'float', 'format', 'frozenset', 'hasattr', 'hash', 'hex', 'int', 'isinstance', 'issubclass', 'iter', 'len', 'list', 'map', 'max', 'min', 'next', 'object', 'oct', 'ord', 'pow', 'range', 'repr', 'reversed', 'round', 'set', 'slice', 'sorted', 'str', 'sum', 'tuple', 'zip' ] return eval( code, globals={ '__builtins__': { builtin : getattr(__builtins__, builtin) for builtin in allowed_builtins } }, locals={}, )  Re: "Why do you want to do this?" I want to be able to filter python objects based on user input. These objects are "payloads" for "sessions" in the application, so allowing users to filter sessions based on arbitrary expressions on payload contents - e.g. len(payload.things) > 12 - would be a useful feature. So my question is: what input string would allow an attacker access data "outside" the eval, i.e. variables in the script or access to the OS?

Answer: Just put a space in: . __ and the standard bypasses work fine. Like this one: [x for x in (). __class__. __base__. __subclasses__() if x. __name__ == 'Quitter'][0]. __init__. __globals__['__builtins__']['__import__']( 'os').system('install ransomware or something')  Seriously, don't use eval. These ad-hoc sanitizers never cover the full attack surface, and even if one miraculously does, the attack surface expands with every Python version.