eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I have a data frame that oscillates between negative and positive values. What I'm looking to do is find the value of the last row, and return a dataframe of all rows above it until the sign changes form negative to positive or vice versa. An example of problem would be something like this import numpy as np import pandas as pd df = pd.DataFrame(np.random.randint(-10,10,size=(100, 1)), columns=list('v'))  creating a dataframe that contains numbers from -10 to 10. What I need to do is find the very last row of "V" and return all rows above it until it switches signs. Running this code gave me the following output : 93, 3 94, 2 95, 0 96, -1 97, -7 98, -2  What im trying to get as new dataframe containing rows 96 - 98, and everything above 95 is not needed or sliced off. Since the sign changes so many times inside the dataframe, im having trouble singling out the last change before the end of the dataframe (row 95) I've tried various types of slicing with iloc and .tail() but haven't had any success. df[(df['v'].tail() < 0).idxmin(): ]  is as close as I've got, however it only returns previous 5 values and sometimes it can take up to 20 index's to change signs. I've tried in various forms data[: (df['v'].iloc[-1] < 0)] but I just cant seem to get exactly what I'm after. A big hangup is the data set is hundreds rows long, so there are too many zero's or sign changes to go about other methods that I can think of, when I need data only after the last sign change. Any help would be greatly appreciated.

Answer: Example Code If you generate a DataFrame randomly, you must provide a seed. I will generate a new DataFrame as an example because you did not provide a seed. import pandas as pd df = pd.DataFrame([1, 2, 3, -4, 0, 1, 2] , columns=list('v'))  df:  v 0 1 1 2 2 3 3 -4 4 0 5 1 6 2   Code s = df['v'].mask(df['v'].eq(0)).ffill() grp = s.mul(s.shift()).le(0).cumsum() out = df[grp.eq(grp.max())]  out:  v 5 1 6 2
Question: Traceback (most recent call last): File "C:\Users\nenuk\OneDrive\Desktop\Creator\32.py", line 556, in <module> driver = uc.Chrome(options=opts) File "C:\Users\nenuk\AppData\Local\Programs\Python\Python312\Lib\site-packages\undetected_chromedriver\__init__.py", line 258, in __init__ self.patcher.auto() File "C:\Users\nenuk\AppData\Local\Programs\Python\Python312\Lib\site-packages\undetected_chromedriver\patcher.py", line 178, in auto self.unzip_package(self.fetch_package()) File "C:\Users\nenuk\AppData\Local\Programs\Python\Python312\Lib\site-packages\undetected_chromedriver\patcher.py", line 287, in fetch_package return urlretrieve(download_url)[0] File "C:\Users\nenuk\AppData\Local\Programs\Python\Python312\Lib\urllib\request.py", line 240, in urlretrieve with contextlib.closing(urlopen(url, data)) as fp: File "C:\Users\nenuk\AppData\Local\Programs\Python\Python312\Lib\urllib\request.py", line 215, in urlopen return opener.open(url, data, timeout) File "C:\Users\nenuk\AppData\Local\Programs\Python\Python312\Lib\urllib\request.py", line 521, in open response = meth(req, response) File "C:\Users\nenuk\AppData\Local\Programs\Python\Python312\Lib\urllib\request.py", line 630, in http_response response = self.parent.error( File "C:\Users\nenuk\AppData\Local\Programs\Python\Python312\Lib\urllib\request.py", line 559, in error return self._call_chain(*args) File "C:\Users\nenuk\AppData\Local\Programs\Python\Python312\Lib\urllib\request.py", line 492, in _call_chain result = func(*args) File "C:\Users\nenuk\AppData\Local\Programs\Python\Python312\Lib\urllib\request.py", line 639, in http_error_default raise HTTPError(req.full_url, code, msg, hdrs, fp) urllib.error.HTTPError: HTTP Error 404: Not Found  How i can solve this problem, last time i re-downloaded everything it worked but right now it doesnt.

Answer:  def fetch_package(self): """ Downloads ChromeDriver from source :return: path to downloaded file """ zip_name = f"chromedriver_{self.platform_name}.zip" if self.is_old_chromedriver: download_url = "%s/%s/%s" % (self.url_repo, self.version_full.vstring, zip_name) else: zip_name = zip_name.replace("_", "-", 1) #download_url = "https://edgedl.me.gvt1.com/edgedl/chrome/chrome-for-testing/%s/%s/%s" download_url = "https://storage.googleapis.com/chrome-for-testing-public/%s/%s/%s" download_url %= (self.version_full.vstring, self.platform_name, zip_name) logger.debug("downloading from %s" % download_url) return urlretrieve(download_url)[0]  You need to change fetch_package(self) in undetected_chromedriver\patcher.py Since the old download_url can not work to get the chrome driver, you should use 'https://storage.googleapis.com/' to replace the 'https://edgedl.me.gvt1.com/edgedl/chrome/'.
Question: I'm new to flyte, trying to do quite simple things at the moment. So I'm trying to to run multiple independant tasks in parallel, which, if I understand the documentation, is exactly the purpose of map_task. However, so far I've been unable to make flyte actually run them in parallel. Any help would be greatly appreciated. Thanks in advance. from flytekit import map_task, task, workflow @task def do_something(value: str) -> str: print(f"launched: {value}", flush=True) time.sleep(60) # fakes long process time return f"{value}-processed" @workflow def do_multiple_things() -> list[str]: values = ["foo", "bar", "baz"] return map_task(do_something)(value=values) 

Answer: Could you give more context please? How are you trying to run it? Is this running in remote (like on a live Flyte backend) or is this a local run? Local runs will not actually do parallel just yet. (Making flytekit execute local runs in parallel is part of a broader project that we have plans for some day, but no definite timeline). A side note however, as of this writing (circa early 2024) there is a newer version of map task that we are steadily moving towards. If you could, I would suggest using this instead. from flytekit.experimental import map_task  At some point the main map task will become this one (though we will keep the experimental import in place for compatibility).
Question: I'm building a report that will show the line numbers of XML elements that match a set of XPaths. I need to support XPath 2.0. Sending the XML to a separate web based processor written in Java or C# is a valid solution, but one I'm avoiding because my entire team works in Python, I want my tool to still work offline, and maintaining another web service is a lot of work. Saxonche supports XPath 2.0. The documentation describes multiple options for enabling line numbers, but never explains how to get the line numbers out once you have enabled them. Here's my code: input_file_path = 'test.xml' # Contents below input_xpath = './/foo' with PySaxonProcessor(license=False) as saxon_proc: # Attempt #1 to enable line numbers saxon_proc.set_configuration_property('l', 'on') doc_builder = saxon_proc.new_document_builder() # Attempt #2 to enable line numbers doc_builder.set_line_numbering(True) xml_tree = doc_builder.parse_xml(xml_file_name=input_file_path) xpath_processor = saxon_proc.new_xpath_processor() xpath_processor.set_context(xdm_item=xml_tree) foo_elements = xpath_processor.evaluate(input_xpath) # Do not see any line numbers on foo_elements in the debugger  I inspected the result of evaluate() in the debugger, but I don't see anything that looks like a line number.  Both PySaxonProcessor and PyDocumentBuilder have a parse_xml() method. In my code I am using PyDocumentBuilder, but I tried both and didn't notice any differences. test.XML <root> <foo>fah</foo> </root>  Apparently there are wrong ways to feed your XML to Saxon that can result in no line numbers, but all of the information I found about that is in other languages. Any ideas about what I am doing wrong?

Answer: I am afraid I can't currently tell there is a way for SaxonC HE, for PE/EE you should be able to use the Saxon XPath extension function saxon:line-number e.g. from saxoncee import PySaxonProcessor with PySaxonProcessor(license=True) as saxon_proc: print(saxon_proc.version) doc_builder = saxon_proc.new_document_builder() doc_builder.set_line_numbering(True) xdm_doc = doc_builder.parse_xml(xml_file_name='sample1.xml') xpath_processor = saxon_proc.new_xpath_processor() xpath_processor.set_context(xdm_item=xdm_doc) xpath_processor.declare_namespace('saxon', 'http://saxon.sf.net/') items = xpath_processor.evaluate('//item') for item in items: xpath_processor.set_context(xdm_item=item) print(item, xpath_processor.evaluate_single('saxon:line-number(.)'))  As I said, I am currently not sure whether there is a way for SaxonC HE, will try to investigate, https://www.saxonica.com/saxon-c/doc12/html/saxonc.html#PyXdmNode, however, doesn't seem to expose any properties similar to the Java API's XdmNode's https://www.saxonica.com/html/documentation12/javadoc/net/sf/saxon/s9api/XdmNode.html#getLineNumber().
Question: In my code (python) I have a class defined as follows: class Properties: def __init__(self,**kwargs): self.otherproperty = kwargs.get("otherproperty") self.channel = kwargs.get("channel") if self.channel not in "A": print("bad channel input - it should be A") pass  In the rest of my code, in various places I calculate and add various attributes to the instance of that class: prop = Properties() prop.channel="B" #lots of calculations here prop.otherproperty= "not a channel"  But when I do it this way, I get an error: TypeError: 'in <string>' requires string as left operand, not NoneType  I already figured out that the following way I have no issues, and the new attribute gets nicely validated, and the user is informed that prop.channel should be different : prop = Properties(otherproperty = "not a channel",channel="B")  But because of several reasons it would be inconvenient to define prop object this way. So how do I validate the object's attributes when I'm adding them step-by-step? And how is such step-by-step object building professionally named?

Answer: You can define a custom setter for the channel property with Python descriptors: class Properties: def __init__(self, **kwargs): self.otherproperty = kwargs.get("otherproperty") self._channel = None # Private attribute for channel @property def channel(self): return self._channel @channel.setter def channel(self, value): if value not in "A": raise ValueError("Bad channel input, it must be a substring of 'A'") self._channel = value  The example usage would be: prop = Properties() prop.channel = "B" # Raises ValueError  This way of gradually adding attributes to an object can be called "lazy object initialization" and is useful if you want to avoid long computations before creating an object. Side note: the above code doesn't raise an error if you set prop.channel = "". So you might want to change the validation to if value != "A": if this was an unintended behavior.
Question: I have a non-empty integer interval [a; b). I want to generate a random non-empty integer subinterval [c; d) (where a <= c and d <= b). The [c; d) interval must be uniformly distributed in the sense that every point in [a; b) must be equally likely to end up in [c; d). I tried generating uniformly distributed c from [a; b - 1), and then uniformly distributed d from [c + 1; b), like this: a = -100 b = 100 N = 10000 cs = np.random.randint(a, b - 1, N) ds = np.random.randint(cs + 1, b)  But when measuring how often each point ends up being sampled, the the distribution is clearly non-unifrom: import numpy as np import matplotlib.pyplot as plt hist = np.zeros(b - a, int) for c, d in zip(cs, ds): hist[c - a:d - a] += 1 plt.plot(np.arange(a, b), hist) plt.show()   How do I do this correctly?

Answer: If you divide the range [a,b) into N sections, and then select one of those sections at random, then the chance of selecting each point is uniform -- 1/N. It doesn't matter how you divide the range. Here's a solution along those lines that uses a pretty uniform selection of division points. from random import randint a, b, N = -100, 100, 1000000 intervals = [] while len(intervals) < N: # divide the range into 3 intervals [a,x), [x,y), and [y,b) # the distributions of the division points don't change the histogram # we're careful here to make sure none of them are empty p1 = randint(a+1,b-2) p2 = randint(a+1,b-2) x = min(p1,p2) y = max(p1,p2)+1 # select one at random intervals.append([(a,x),(x,y),(y,b)][randint(0,2)])
Question: I'm currently using Open3D in Python to voxelize meshes, and I've come across a peculiar behavior during the process. When voxelizing a box mesh, it appears that the resulting voxels only align along the lines of the triangles instead of filling the entire surface area of the triangle faces, particularly noticeable on the box's walls. Is this behavior expected in Open3D's voxelization process, or is there a misunderstanding in my approach? I attempted voxelization on a box mesh, expecting the voxel grid to uniformly fill all the triangles of the mesh, that comprises the faces of the box. However, upon visualization, I noticed that while the top and bottom faces are adequately filled, the walls of the box exhibit voxelization primarily along the triangle edges. import open3d as o3d import numpy as np new_room = o3d.geometry.TriangleMesh.create_box(10,10,10) lines = [] for triangle in np.array(new_room.triangles): lines.extend([(triangle[0], triangle[1]), (triangle[1], triangle[2]), (triangle[2], triangle[0])]) line_set = o3d.geometry.LineSet() vertices = np.array(new_room.vertices) line_set.points = o3d.utility.Vector3dVector(vertices) line_set.lines = o3d.utility.Vector2iVector(lines) voxel_grid = o3d.geometry.VoxelGrid.create_from_triangle_mesh(new_room, voxel_size=0.5) o3d.visualization.draw_geometries([new_room, line_set, voxel_grid])  Upon visualization, only the top and bottom faces are adequately filled, while the walls show voxelization predominantly along the diagonals. I'm using Open3D version v0.17.0.

Answer: After encountering an issue with Open3D's voxelization process, I discovered that the problem had been raised and addressed on GitHub in this thread. The issue was resolved in Open3D version v0.18.0. However, when attempting to update the package using pip, I encountered difficulties as pip couldn't find the correct compatible tag on PyPI with the desired version. This issue arose due to using an Apple Mac M1, which has some compatibility issues with package tags. Thankfully, I found a solution documented here: SYSTEM_VERSION_COMPAT=0 pip install --upgrade open3d  Running this command resolved the compatibility issue and allowed me to successfully update Open3D to the required version. The voxelization is working perfectly now.
Question: Suppose I have four (N,) vectors A, B, C, and D: import numpy as np N = 100 A = 1*np.ones(N) # => array([1,1,1,...]) B = 2*np.ones(N) # => array([2,2,2,...]) C = 3*np.ones(N) # => array([3,3,3,...]) D = 4*np.ones(N) # => array([4,4,4,...])  In my application, these are each an element of a matrix and the matrix varies over time (although in this example each is constant in time). I want a matrix with shape (N,2,2) such that I have a 2x2 matrix for each time step, like [[a,b],[c,d]], extended along axis 0. Is there a nicer way to stack these into that kind of shape than what I have below? My solution: A_ = np.reshape(A, (N,1,1)) B_ = np.reshape(B, (N,1,1)) C_ = np.reshape(C, (N,1,1)) D_ = np.reshape(D, (N,1,1)) AB = np.concatenate((A_, B_),axis=2) CD = np.concatenate((C_, D_),axis=2) ABCD = np.concatenate((AB,CD),axis=1)  This gives: >>> ABCD array([[[1., 2.], [3., 4.]], [[1., 2.], [3., 4.]], [[1., 2.], [3., 4.]], ... ... [[1., 2.], [3., 4.]], [[1., 2.], [3., 4.]]])  As desired. It's just difficult to understand what it's doing, so I'm wondering if there's a better way.

Answer: Concatenate as a 3D intermediate with dstack, then you'll be in the ideal order to perform a simple reshape: np.dstack([A, B, C, D]).reshape(N, 2, 2)  Output: array([[[1., 2.], [3., 4.]], [[1., 2.], [3., 4.]], [[1., 2.], [3., 4.]], ..., [[3., 4.], [3., 4.]]])  Intermediate: np.dstack([A,B,C,D]).shape # (1, 100, 4)
Question: So I have many dataframes coming in that need to be exploded. they look something like this: df = pd.DataFrame({'A': [1, [11,22], [111,222]], 'B': [2, [33,44], float('nan')], 'C': [3, [55,66], [333,444]], 'D': [4, [77,88], float('nan')] })  +-----------+---------+-----------+---------+ | A | B | C | D | +-----------+---------+-----------+---------+ | 1 | 2 | 3 | 4 | +-----------+---------+-----------+---------+ | [11,22] | [33,44] | [55,66] | [77,88] | +-----------+---------+-----------+---------+ | [111,222] | NaN | [333,444] | NaN | +-----------+---------+-----------+---------+  Typically if a column couldn't be exploded I'd just remove it from the column list like so: colList = df.columns.values.tolist() colList.remove("B") colList.remove("D") df = df.explode(colList)  But that would leave me with a dataframe that looks like: +-----+---------+-----+---------+ | A | B | C | D | +-----+---------+-----+---------+ | 1 | 2 | 3 | 4 | +-----+---------+-----+---------+ | 11 | [33,44] | 55 | [77,88] | +-----+---------+-----+---------+ | 22 | [33,44] | 66 | [77,88] | +-----+---------+-----+---------+ | 111 | NaN | 333 | NaN | +-----+---------+-----+---------+ | 222 | NaN | 444 | NaN | +-----+---------+-----+---------+  I still need to explode those columns (B and D in example), but if I do, it'll throw an error due to the nulls. How can I successfully explode dataframes with this sort of problem?

Answer: One option could be to explode each column separately and deduplicate the index before concat: def explode_dedup(s): s = s.explode() return s.set_axis( pd.MultiIndex.from_arrays([s.index, s.groupby(level=0).cumcount()]) ) out = pd.concat({c: explode_dedup(df[c]) for c in df}, axis=1)  Output:  A B C D 0 0 1 2 3 4 1 0 11 33 55 77 1 22 44 66 88 2 0 111 NaN 333 NaN 1 222 NaN 444 NaN  To explode a subset of the columns: cols = ['C', 'D'] others = df.columns.difference(cols) out = (df[others] .join(pd.concat({c: explode_dedup(df[c]) for c in cols}, axis=1) .droplevel(-1) )[df.columns] )  Output:  A B C D 0 1 2 3 4 1 [11, 22] [33, 44] 55 77 1 [11, 22] [33, 44] 66 88 2 [111, 222] NaN 333 NaN 2 [111, 222] NaN 444 NaN  Reproducible input: df = pd.DataFrame({'A': [1, [11,22], [111,222]], 'B': [2, [33,44], float('nan')], 'C': [3, [55,66], [333,444]], 'D': [4, [77,88], float('nan')] })
Question: My input is this dataframe : df = pd.DataFrame( { 'ID': ['ID001', 'ID002', 'ID003'], 'DATE': ['24/12/2023', '01/02/2024', '12/02/2024'], } ) df['DATE'] = pd.to_datetime(df['DATE'], dayfirst=True) print(df) ID DATE 0 ID001 2023-12-24 1 ID002 2024-02-01 2 ID003 2024-02-12  I'm trying to duplicate the rows for each id N times with N being the number of weeks between the column DATE and the current date. At the end, each last row for a given id will have the week of the column DATE. For that I made the code below but it gives me a wrong output : number_of_weeks = (pd.Timestamp('now') - df['DATE']).dt.days // 7 final = df.copy() final['YEAR'] = final['DATE'].dt.isocalendar().year final['WEEK'] = final['DATE'].dt.isocalendar().week final['WEEKS'] = (pd.Timestamp('now') - df['DATE']).dt.days // 7 for index, row in final.iterrows(): for i in range(1, row['WEEKS'] + 1): final.loc[i, 'WEEK'] = i final = final.ffill().drop(columns='WEEKS') print(final) ID DATE YEAR WEEK 0 ID001 2023-12-24 2023 51 1 ID002 2024-02-01 2024 1 2 ID003 2024-02-12 2024 2 3 ID003 2024-02-12 2024 3 4 ID003 2024-02-12 2024 4 5 ID003 2024-02-12 2024 5 6 ID003 2024-02-12 2024 6 7 ID003 2024-02-12 2024 7  Have you guys encountered a similar problem ? I'm open to any suggestion. My expected output is this :  ID DATE YEAR WEEK 0 ID001 24/12/2023 2023 51 1 ID001 24/12/2023 2023 52 2 ID001 24/12/2023 2024 1 3 ID001 24/12/2023 2024 2 4 ID001 24/12/2023 2024 3 5 ID001 24/12/2023 2024 4 6 ID001 24/12/2023 2024 5 7 ID001 24/12/2023 2024 6 8 ID001 24/12/2023 2024 7 ################################## 9 ID002 01/02/2024 2024 5 10 ID002 01/02/2024 2024 6 11 ID002 01/02/2024 2024 7 ################################## 12 ID003 12/02/2024 2024 7 

Answer: I would use periods for that and Index.repeat: # compute exact difference in weeks n_weeks = (df['DATE'].dt.to_period('W') .rsub(pd.Timestamp('now').to_period('W')) .apply(lambda x: x.n) ) # repeat rows, add incrementing days, convert to week number out = (df.loc[df.index.repeat(n_weeks+1)] .assign(WEEK=lambda d: df['DATE'] .add(pd.to_timedelta(d.groupby(level=0).cumcount()*7, unit='D')) .dt.isocalendar().week ) )  Output:  ID DATE WEEK 0 ID001 2023-12-24 51 0 ID001 2023-12-24 52 0 ID001 2023-12-24 1 0 ID001 2023-12-24 2 0 ID001 2023-12-24 3 0 ID001 2023-12-24 4 0 ID001 2023-12-24 5 0 ID001 2023-12-24 6 0 ID001 2023-12-24 7 1 ID002 2024-02-01 5 1 ID002 2024-02-01 6 1 ID002 2024-02-01 7 2 ID003 2024-02-12 7
Question: I am leveraging the function .to_dicts() in my testing the resulting list of dictionaries has the same order as the dataframe. df = pl.DataFrame(data={ "name": ["Alice", "Bob", "Charlie"], "age": [30, 25, 35], "city": ["New York", "London", "Paris"], }).sort(by=['name'], descending=True) data = df.to_dicts() for record in data: print(record)  Results in: {'name': 'Charlie', 'age': 35, 'city': 'Paris'} {'name': 'Bob', 'age': 25, 'city': 'London'} {'name': 'Alice', 'age': 30, 'city': 'New York'}  Does anyone know if that is guaranteed to happen every time?

Answer: TLDR. Yes. This can be seen by inspecting this underlying rust function, which maps over the indices of the DataFrame in order.
Question: I am trying to write unit tests for my python project using pytest. I have a module from another repo called sql_services. Within sql_services there is a function called read_sql that I am trying to mock. So far, I have only been able to mock the function if I import the module like import sql_services  and invoke sql_services.read_sql but I have been unable to mock it if I import the function like from sql_services import read_sql  This is a problem because my codebase uses the latter method for importing the functions Here is the function I am trying to write a unit test for: from sql_services import read_sql def foo(): df = read_sql("SELECT * FROM schema.table") return df  Here is the unit test file I have so far: import pytest import unittest.mock as mock import pandas as pd import sql_services from entry import foo @mock.patch("sql_services.read_sql") def test_read_sql(mock_read_sql): mock_read_sql.return_value = pd.DataFrame() df = sql_services.read_sql("SELECT * FROM schema.table") assert df.empty @mock.patch("sql_services.read_sql") def test_do_a_read(mock_read_sql): mock_read_sql.return_value = pd.DataFrame() df = foo() assert df.empty  The first test passes and the second fails because it actually reads the data frame from the database. Is there any way I can mock the function from within foo without refactoring my entire codebase?

Answer: You just patch the correct name: @mock.patch("entry.read_sql") def test_read_sql(mock_read_sql): mock_read_sql.return_value = pd.DataFrame() df = sql_services.read_sql("SELECT * FROM schema.table") assert df.empty]  entry.foo is using the global variable entry.read_sql to access the function you want to mock, not the global variable sql_servies.read_sql.