eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I installed a third-party driver and firmware to use my PS 4 Camera on PC and I need it to do a object detection project. Everything works fine on camera on Windows (the screen resolution looks buggy but this is simply how Windows read Sony data format). Here is how it looks (I have a red light in the room):  -- When I open the same cam on OpenCV the image was looking very bright, or it looks very low quality. Here is how the image looks in OpenCV when saved with imwrite():  In the first image you can clearly read ALIENWARE under the monitor, in the second image is clearly granulated. I wrote the simplest code just to read and show the cam: import cv2 #read webcam = cv2.VideoCapture(0) #visualize while True: ret, frame = webcam.read() cv2.imshow('webcam', frame) if cv2.waitKey(40) & 0xFF == ord('q'): break webcam.release() cv2.destroyAllWindows()  How can I fix this?

Answer: I solved the problem! With ffmpeg/ffplay I figured out that this cam can only support certain frame/resolution combinations: [dshow @ 000002e52f473900] pixel_format=yuyv422 min s=3448x808 fps=8 max s=3448x808 fps=60.0002 (tv, bt470bg/bt709/unknown, topleft) [dshow @ 000002e52f473900] pixel_format=yuyv422 min s=1748x408 fps=8 max s=1748x408 fps=120 (tv, bt470bg/bt709/unknown, topleft) [dshow @ 000002e52f473900] pixel_format=yuyv422 min s=898x200 fps=30 max s=898x200 fps=240.004 (tv, bt470bg/bt709/unknown, topleft)  it appears that if you try to lower/increase the fps while keeping the same resolution it adjust the resolution itself to the nearest supported resolution but keeping the said framerate (if between min-max interval). That happens on OpenCV as well. Now, with  width = 3448 height = 808 cap.set(cv2.CAP_PROP_FRAME_WIDTH, width) cap.set(cv2.CAP_PROP_FRAME_HEIGHT, height) fps = 60 cap.set(cv2.CAP_PROP_FPS, fps)  it outputs this image: much better image Thank you all for your help!
Question: In the langchain wiki of FAISS, https://python.langchain.com/v0.2/docs/integrations/vectorstores/faiss/, it only talks about saving indexes to files. db.save_local("faiss_index") new_db = FAISS.load_local("faiss_index", embeddings) docs = new_db.similarity_search(query)  How can I save the indexes to databases, such that we can organize and concurrently access multiple indexes? Searched online but could not get much info on this. Can FAISS be used with any kind of distributed databases?

Answer: In fact, FAISS is considered as an in-memory database itself in order to vector search based on similarity that you can serialize and deserialize the indexes using functions like write_index and read_index within the FAISS interface directly or using save_local and load_local within the LangChain integration which typically uses the pickle for serialization. If you need to store serialized files, you could manually save them in a NoSQL database like MongoDB as binary data, and then deserialize and retrieve them when needed, however, it is not the best practice! If you are looking for a vector database that is not in-memory and capable in a scalable system, you might want to consider using Milvus which is designed for this purpose.
Question: I am building an MPC optimization (IMODE = 6) in Python GEKKO with multiple state (SV) and manipulated (MV) variables. Inside my problem, I would like to define ramping constraints of the following kind: |x(tk) - x(tk-1)| <= Cx for the SV, where Cx is a constant, tk is the current time step and tk-1 is the previous 0.9 <= u(tk) / u(tk-1) <= 1.1 for the MV I tried using the build-in tuning parameter DCOST for the MV, but as far as I understand it suggests using a constant value, which is not my case. Alternatively I tried solving the model iteratively with 1 time step per iteration and taking the values of the previous iteration as starting values for the next one, but it doesn't seem to work. Any clue of how to implement this or an example simple model implementation with such constraints would be of great help.

Answer: For the constraint |x(tk) - x(tk-1)| <= Cx for a state variable, create a new variable dx that is the defined as the derivative of the variable x and set a constraint on the ramp rate. # State Variable x = m.SV(value=0,ub=2) dx = m.SV(value=0,lb=-0.5,ub=0.5) # Process model m.Equation(5*x.dt() == -x + 3*u) m.Equation(dx == x.dt())  For the constraint 0.9 <= u(tk) / u(tk-1) <= 1.1 for the MV, set up a delay variable and constrain the value of u to be no more than a 10% change from the prior value. u = m.MV(value=0.8, lb=0.1, ub=10) # avoid divide-by-zero u.STATUS = 1 # allow optimizer to change ud = m.Var(value=0.8) m.delay(u,ud,1) # delay of 1 m.Equation(u<=1.1*ud) m.Equation(u>=0.9*ud)  If the MV value u ever reaches zero, then the value won't be allowed to change further. It is generally safer to use something like DMAX or DMAXHI/DMAXLO to configure the rate of change of the MV. Below is a complete and minimal example that demonstrates the two constraint methods.  from gekko import GEKKO import numpy as np import matplotlib.pyplot as plt m = GEKKO() m.time = np.linspace(0,10,21) # Manipulated variable u = m.MV(value=0.8, lb=0.1, ub=10) # avoid divide-by-zero u.STATUS = 1 # allow optimizer to change #u.DCOST = 0.1 # smooth out MV movement #u.DMAX = 0.5 # limit MV rate of change ud = m.Var(value=0.8) m.delay(u,ud,1) # delay of 1 m.Equation(u<=1.1*ud) m.Equation(u>=0.9*ud) # State Variable x = m.SV(value=0,ub=2) dx = m.SV(value=0,lb=-0.5,ub=0.5) # Process model m.Equation(5*x.dt() == -x + 3*u) m.Equation(dx == x.dt()) # Objective m.Maximize(x) m.options.IMODE = 6 # control m.solve(disp=False) plt.figure(figsize=(6,3.5)) plt.subplot(2,1,1) plt.step(m.time,u.value,'b-',label='MV Optimized') plt.legend(); plt.ylabel('Input') plt.subplot(2,1,2) plt.plot(m.time,x.value,'r--',label='SV Response') plt.ylabel('State'); plt.xlabel('Time') plt.legend(); plt.tight_layout() plt.show()
Question: I have a simple graph with the attributes height and area import networkx as nx nodes_list = [("A", {"height":10, "area":100}), ("B", {"height":12, "area":200}), ("C", {"height":8, "area":150}), ("D", {"height":9, "area":120})] G = nx.Graph() G.add_nodes_from(nodes_list) edges_list = [("A","B"), ("B","C"), ("C","D")] G.add_edges_from(edges_list) nx.draw(G, with_labels=True, node_color="red")   I want to contract nodes and update the attributes with the average of height and the sum of area of the contracted nodes. There must be some easier way than: H = nx.contracted_nodes(G, "B","C") #print(H.nodes["B"]) #{'height': 12, 'area': 200, 'contraction': {'C': {'height': 8, 'area': 150}}} #Calc the average of node B and C's heights new_height = (H.nodes["B"]["height"] + H.nodes["B"]["contraction"]["C"]["height"])/2 #10 #Calc the sum of of node B and C's areas new_area = H.nodes["B"]["area"] + H.nodes["B"]["contraction"]["C"]["area"] #Drop old attributes del(H.nodes["B"]["height"], H.nodes["B"]["area"], H.nodes["B"]["contraction"]) nx.set_node_attributes(H, {"B":{"height":new_height, "area":new_area}}) print(H.nodes["B"]) #{'height': 10.0, 'area': 350}  Can I get networkx to average height and sum area, every time I contract multiple nodes?

Answer: I don't think there is a builtin way, especially since you have custom aggregation functions. Here is an alternative way, using a custom function: def contract_merge(G, n1, n2): H = nx.contracted_nodes(G, n1, n2) d_n1 = H.nodes[n1] d_n2 = d_n1.pop('contraction')[n2] d_n1['height'] = (d_n1['height']+d_n2['height'])/2 d_n1['area'] = d_n1['area']+d_n2['area'] nx.set_node_attributes(H, {n1: d_n1}) return H H = contract_merge(G, 'B', 'C') print(H.nodes(data=True))  Output: NodeDataView({'A': {'height': 10, 'area': 100}, 'B': {'height': 10.0, 'area': 350}, 'D': {'height': 9, 'area': 120}})
Question: In Python, is Type[Union[A, B, C]] the same as Union[Type[A], Type[B], Type[C]]? I think they are equivalent, and the interpreter seems to agree. ChatGPT (which, from my past experience, tends to be wrong with this type of questions) disagrees so I was wondering which one is more correct. To be clear: I want a union of the types of A, B or C, not a union of instances of A, B or C. The first option is shorter and IMHO more readable. In other words, given these definitions: class A: pass class B: pass class C: pass def foo(my_class: Type[Union[A, B, C]]): pass  I expect this usage to be correct: foo(A) # pass the class A itself  But not this one: foo(A()) # pass an instance of A 

Answer: They are the same. You can use reveal_type and check. class A: ... class B: ... class C: ... def something(a: type[A | B | C]) -> None: reveal_type(a) def something_else(b: type[A] | type[B]| type[C]) -> None: reveal_type(b)  mypy main.py main.py:6: note: Revealed type is "Union[type[main.A], type[main.B], type[main.C]]" main.py:9: note: Revealed type is "Union[type[main.A], type[main.B], type[main.C]]"  pyright main.py main.py:6:17 - information: Type of "a" is "type[A] | type[B] | type[C]" main.py:9:17 - information: Type of "b" is "type[A] | type[B] | type[C]"  Edit: Older syntax from typing import Union, Type class A: ... class B: ... class C: ... def something(a: type[A | B | C]) -> None: reveal_type(a) def something_else(b: type[A] | type[B]| type[C]) -> None: reveal_type(b) def old_something(c: Type[Union[A, B, C]]) -> None: reveal_type(c) def old_something_else(d: Union[Type[A], Type[B], Type[C]]) -> None: reveal_type(d)  mypy main.py main.py:7: note: Revealed type is "Union[type[main.A], type[main.B], type[main.C]]" main.py:10: note: Revealed type is "Union[type[main.A], type[main.B], type[main.C]]" main.py:13: note: Revealed type is "Union[type[main.A], type[main.B], type[main.C]]" main.py:16: note: Revealed type is "Union[type[main.A], type[main.B], type[main.C]]"  pyright main.py main.py:7:17 - information: Type of "a" is "type[A] | type[B] | type[C]" main.py:10:17 - information: Type of "b" is "type[A] | type[B] | type[C]" main.py:13:17 - information: Type of "c" is "type[A] | type[B] | type[C]" main.py:16:17 - information: Type of "d" is "type[A] | type[B] | type[C]"
Question: I have a pandas dataframe with a column. id text_col 1 Was it Accurate?: Yes\n\nReasoning: This is a sample : text 2 Was it Accurate?: Yes\n\nReasoning: This is a :sample 2 text 3 Was it Accurate?: No\n\nReasoning: This is a sample: 1. text  I have to break the text_col into two columms "Was it accurate?" and "Reasoning" The final dataframe should look like: id Was it Accurate? Reasoning 1 Yes This is a sample : text 2 Yes This is a :sample 2 text 3 No This is a sample: 1. text  The text values can have multiple : "colons" in it I tried splitting the text_col using "\n\nReasoning:" but did'nt get desired result.It is leaving out the text after second colon (:) df[['Was it Accurate?', 'Reasoning']] = df['text_col'].str.extract(r'Was it Accurate\?: (Yes|No)\n\nReasoning: (.*)') Edit:  I applied the function on the LLM_response column of my sample_100 dataframe. and printed the first row. if you see closely the sample_100.iloc[0]['Reasoning'] has stripped off all the text after : Temp dict obj to test on: {'id_no': [8736215], 'Notes': [' Temp Notes Sample xxxxxxxxxxxxx [4/21/23, 2:10 PM] Work started -work complete-'], 'ProblemDescription': ['Sample problem description xxxxxxxxxxxxxxxxxxxxxxxx'], 'LLM_response': ['Accurate & Understandable: Yes\n\nReasoning: The Technician notes are accurate and understandable as:\n1) The technician provided detailed steps on how they addressed the mold issue by removing materials, treating surfaces, priming, and painting them.\n2) Additionally, even though there was non-repair related information (toilet repairs), the main issue of mold growth was addressed.\n3) The process described logically follows the process for remedying a mold issue, which aligns with the problem description.'], 'Accurate & Understandable': ['Yes'], 'Reasoning': ['The Technician notes are accurate and understandable as:']} 

Answer: The issue is not due to colons, but to newlines in your sample text. Those are not matched by . by default. You should add the re.DOTALL flag. Example: import re import pandas as pd df = pd.DataFrame({'id': [1, 2, 3], 'text_col': ['Was it Accurate?: Yes\n\nReasoning: This is a sample: text', 'Was it Accurate?: Yes\n\nReasoning: This is a sample:\n with newline', 'Was it Accurate?: No\n\nReasoning: This is a sample text']}) df[['Was it Accurate?', 'Reasoning']] = (df['text_col'] .str.extract(r'Was it Accurate\?: (Yes|No)\n\nReasoning: (.*)', flags=re.DOTALL) )  Output:  id text_col Was it Accurate? Reasoning 0 1 Was it Accurate?: Yes\n\nReasoning: This is a sample: text Yes This is a sample: text 1 2 Was it Accurate?: Yes\n\nReasoning: This is a sample:\n with newline Yes This is a sample:\n with newline 2 3 Was it Accurate?: No\n\nReasoning: This is a sample text No This is a sample text
Question: I was solving a problem on HackerEarth which is as follows:  You are provided an array A of size N that contains non-negative integers. Your task is to determine whether the number that is formed by selecting the last digit of all the N numbers is divisible by 10. Print "Yes" if possible and "No" if not.  Here's what I did: size = int(input()) integers = input() integers = [i for i in integers.split(' ')] last_digit = [i[-1] for i in integers] number = int(''.join(last_digit)) if number % 10 == 0: print("Yes") else: print("No")  The code seems correct to me, but the problem is that the online HackerEarth platform throws a runtime error NZEC, about which I have no idea. So, I want to know why(and what kind of) error occured here and what is the problem with my code.

Answer: https://help.hackerearth.com/hc/en-us/articles/360002673433-types-of-errors Hackerearth says  For interpreted languages like Python, NZEC will usually mean   Usually means that your program has either crashed or raised an uncaught exception  Many runtime errors  Usage of an external library that is not used by the judge    For your case: number = int(''.join(last_digit))  number is becoming a very very big number which is the reason you are getting NZEC error (they have limited memory and time to execute the problem) little modified code: size = int(input()) integers = input() integers = [i for i in integers.split(' ')] last_digit = [i[-1] for i in integers] number = ''.join(last_digit) # keeping the number as string if number[-1] == '0': # checking the last digit of the string to be 0 print("Yes") else: print("No")
Question: I am trying to apply a function to a Dataframe column (series) that retrieves the day of the week based on the timestamps in the column. However, I am being thrown the following exception, even though the Polars docs include documentation for polars.Expr.apply. AttributeError: 'Expr' object has no attribute 'apply'.  My goal is to create a new column of day names using the following code where the alertTime column is of dtype datetime64: def get_day(dt_obj): days_of_week = ['Monday','Tuesday','Wednesday','Thursday','Friday','Saturday','Sunday'] return days_of_week[dt_obj.weekday()] # Get the day of the week from the timestamp df = df.with_columns( pl.col('alertTime').apply(get_day, return_dtype=pl.Utf8).alias('day_of_week') )  Could anyone help with where I might be going wrong?

Answer: apply was renamed to .map_elements() some time ago. Previous versions printed a deprecation warning, but it was eventually removed after a grace period. You're likely looking at the docs for an older version of Polars, but there is a "version switcher" on the docs site:    As for the actual task, you can also do it natively using .dt.to_string() import datetime import polars as pl pl.select( pl.lit(str(datetime.datetime.now())) .str.to_datetime() .dt.to_string("%A") )  shape: (1, 1) ┌─────────┐ │ literal │ │ --- │ │ str │ ╞═════════╡ │ Tuesday │ └─────────┘
Question: Python 3.10.4 is embedded into cpp application. I'm trying to import sklearn library which is installed at custom location using pip --target. sklearn custom path (--target path) is appended to sys.path. Below is a function from the script which just prints the version information. Execution using Command Line works well as shown below. python3.10 -c 'from try_sklearn import *; createandload()'  Output [INFO ] [try_sklearn.py:23] 3.10.4 (main, Aug 4 2023, 01:24:50) [GCC 4.8.5 20150623 (Red Hat 4.8.5-16)] [INFO ] [try_sklearn.py:24] sklearn /users/xxxx/temp/python/scikit-learn/sklearn/__init__.py Version = 1.5.1  The same script when called using CPP, hangs at import sklearn  Other libraries like pandas, numpy etc works without any issues.

Answer: https://github.com/scipy/scipy/issues/21189 Looks like Scipy and Numpy do not support Embedded python
Question: I'm having an abstract class in Python like this class Sinker(ABC): @abstractmethod def batch_write(self, data) -> None: pass  The concrete class is expected to write data to some cloud databases. My intention of writing abstract class this way was to make sure the concrete classes always implement the batch_write() method. However, I'm not sure what type hints I should put in for data because one of my concrete class is expecting List[str] while another concrete class is expecting List[dict]. Here are just some options popped up in my head so far.  Any List[Any] List[str|dict]  What would be a better way to do so? Is there a style guide for this situation? I tried out all three options and since they're just type hints, it won't cause me any trouble. But I just want to make sure I'm writing codes that are aligned with corresponding best practices in OOP if there's any.

Answer: You can make Sinker a generic class so the type of items in the list passed to batch_write can be parameterized: from abc import ABC, abstractmethod class Sinker[T](ABC): @abstractmethod def batch_write(self, data: list[T]) -> None: ... class StrSinker(Sinker[str]): def batch_write(self, data: list[str]) -> None: pass class DictSinker(Sinker[dict]): def batch_write(self, data: list[dict]) -> None: pass  Demo with Pyright This way, the type checker would be able to spot an incorrect type hint for you. For example, with: class StrSinker(Sinker[str]): def batch_write(self, data: list[int]) -> None: pass  Pyright would produce the following complaint: Method "batch_write" overrides class "Sinker" in an incompatible manner Parameter 2 type mismatch: base parameter is type "list[str]", override parameter is type "list[int]" "list[str]" is incompatible with "list[int]" Type parameter "_T@list" is invariant, but "str" is not the same as "int" Consider switching from "list" to "Sequence" which is covariant (reportIncompatibleMethodOverride)  Demo with Pyright
Question: For each line of a text file I want to do heavy calculations. The amount of lines can be millions so I'm using multiprocessing: num_workers = 1 with open(my_file, 'r') as f: with multiprocessing.pool.ThreadPool(num_workers) as pool: for data in pool.imap(my_func, f, 100): print(data)  I'm testing interactively hence ThreadPool() (will be replaced in final version). For map or imap documentation says:  This method chops the iterable into a number of chunks which it submits to the process pool as separate tasks  Since my opened file is an iterable (each iteration is a line) I expect this to work but it breaks up lines in the middle. I made a generator that returns lines as expected, but I want to understand why the generator is needed at all. Why isn't the chunking happening on line boundaries? UPDATE: to clarify if I use this generator: def chunked_reader(file, chunk_size=100): with open(file, 'r') as f: chunk = [] i = 0 for line in f: if i == chunk_size: yield chunk chunk = [] i = 0 i += 1 chunk.append(line) # yield final chunk yield chunk  the calculation works, returns expected values. If I use the file object directly, I get errors somehow indicating the chunking is not splitting on line boundaries. UPDATE 2: it's chunking the line into each individual character and not line by line. I can also take another environment with python 3.9 and that has the same behavior. So it has been like this for some time and seems to be "as designed" but not very intuitive. UPDATE 3: To clarify on the marked solution, my misunderstanding was that chunk_size will send a list of data to process to my_func. Internally my_func iterates over the passed in data. But since my assumption wrong and each line gets send separately to my_func regardless of chunk_size, the internal iteration was iterating over the string, not as I expected list of strings.

Answer: imap chunking is an implementation detail, your function will be called with a single element of the iterable regardless of the chunk size. for line in file: result = my_func(line)  Becomes for result in pool.imap(my_func, file, some_chunk_size): pass  It is syntactically the same as builtins.map without the chunk size, but people don't really use it either. The chunking size controls how many elements are packed in a list before being pickled and sent over a pipe, the chunking process is transparent to your code, your function wil always be called with a single line regardless of the chunk size, you should play around with the chunk size and see if it improves performance in your case. Note that imap interally has a for loop, so it should work exactly like a simple for loop on the iterated object.
Question: I'm working on a small Python program for myself and I need an algorithm for fast multiplication of a huge array with prime powers (over 660 000 numbers, each is 7 digits). The result number is over 4 millions digits. Currently I'm using math.prod, which calculates it in ~10 minutes. But that's too slow, especially if I want to increase amount of numbers. I checked some algorithms for faster multiplications, for example the Schönhage–Strassen algorithm and Toom–Cook multiplication, but I didn't understand how they work or how to implement them. I tried some versions that I've found on the internet, but they're not working too well and are even slower. I wonder if someone knows how to multiply these amounts of numbers faster, or could explain how to use some math to do this?

Answer: There are two keys to making this fast. First, using the fastest mult implementation you can get. For "sufficiently large" multiplicands, Python's Karatsuba mult is O(n^1.585). The decimal module's much fancier NTT mult is more like O(n log n). But fastest of all is to install the gmpy2 extension package, which wraps GNU's GMP library, whose chief goal is peak speed. That has essentially the same asymptotics as decimal mult, but with a smaller constant factor. Second, the advanced mult algorithms work best when multiplying two large ints of about the same size (number of bits). You can leave that to luck, or, as below, you can force it by using a priority queue and, at each step, multiplying the "two smallest" partial products remaining. from gmpy2 import mpz from heapq import heapreplace, heappop, heapify # Assuming your input ints are in `xs`. mpzs = list(map(mpz, xs)) heapify(mpzs) for _ in range(len(mpzs) - 1): heapreplace(mpzs, heappop(mpzs) * mpzs[0]) assert len(mpzs) == 1 # the result is mpzs[0]  That's the code I'd use. Note that the cost of recursion (which this doesn't use) is trivial compared to the cost of huge-int arithmetic. Heap operations are more expensive than recursion, but still relatively cheap, and can waaaaay more than repay their cost if the input is in an order such that the "by luck" methods aren't lucky enough.