eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: Please consider the following class A(object): def foo(self): pass a = A() # accessing a's foo seems consistent init1 = a.foo init2 = a.foo assert init1 == init2 assert id(a.foo) == id(a.foo) # Or is it? foos= [a.foo for i in range(10)] ids = [id(foo) for foo in foos] for i, id_ in enumerate(ids): for j, id__ in enumerate(ids): if i != j: assert id_ != id__  It appears id(a.foo) can equal to id(a.foo) and can not equal to id(a.foo), but I can't understand when it is the same and when it isn't. Please explain what's going on here.

Answer: The transformation of a function to an instance method happens every time the instance method attribute is accessed, so the id should be different every time, at least in the sense that a new object has been created. (see Python Data Model for details about method transformation and descriptors) The problem is doing this: assert id(a.foo) == id(a.foo)  There are times the Python garbage collector can work so fast that even within a single expression, two different objects can have the same id, because the object has already been reclaimed once id() is done with it. If you do this: assert id(init1) == id(init2)  you'll see that they in fact have different ids.  Update: to address the question about why init1 == init2 is True: init1 and init2 are method wrapper objects that refer to the same function in the same class, so the method wrapper's __eq__() considers them equal.
Question: I have 3D vertices from a third-party data source. The plotly Volume object expects all the coordinates as 1D lists. The examples on their website use the mgrid function to populate the 3D space into the flatten function to get the 1D lists of each axis. https://plotly.com/python/3d-volume-plots/ I don't understand why my approach produces an empty plot. coords is my list of vertices in the shape of (N, 3). See the following code snippet that draws random coordinates, sorts them, but results in an empty render. X = np.random.uniform(0, 1, 30000) Y = np.random.uniform(0, 1, 30000) Z = np.random.uniform(0, 1, 30000) coords = np.dstack((X.flatten(), Y.flatten(), Z.flatten()))[0] sort_idx = np.lexsort((coords[:, 0], coords[:, 1], coords[:, 2])) coords = coords[sort_idx] X=coords[:, 0] Y=coords[:, 1] Z=coords[:, 2] V = np.sin(X) * np.sin(Y) + Z fig = go.Figure(data=go.Volume( x=X, y=Y, z=Z, value=V, isomin=np.min(Z), isomax=np.max(Z), opacity=0.1, # needs to be small to see through all surfaces surface_count=20, # needs to be a large number for good volume rendering colorscale='Spectral', reversescale=True )) fig.show()  Update: It seems like plotly expects the coordinates to be sorted. X, Y, Z = np.mgrid[-50:50:40j, -50:50:40j, -8:8:10j] coords = np.dstack((X.flatten(), Y.flatten(), Z.flatten()))[0] np.random.shuffle(coords)  Shuffling the list like this and plugging coords into the code above produces an empty Volumn render. I now tried to sort my data points, but I still get an empty render. How can I share my dataset? npfile, but where should I host it? sort_idx = np.lexsort((coords[:, 0], coords[:, 1], coords[:, 2])) coords = coords[sort_idx]  Update 2: Using a uniform random distribution to generate the coordinates results in a vertex list that seems to be not processable by plotly even after sorting. X = np.random.uniform(0, 1, 30000) Y = np.random.uniform(0, 1, 30000) Z = np.random.uniform(0, 1, 30000) coords = np.dstack((X.flatten(), Y.flatten(), Z.flatten()))[0] 

Answer: I have not read the documentation closely enough.  Draws volume trace between iso-min and iso-max values with coordinates given by four 1-dimensional arrays containing the value, x, y and z of every vertex of a uniform or non-uniform 3-D grid.  The input has to be in the form of an orthogonal grid; thus a cloud of random points is not processable by Volumn. Non-uniformity means in this case that along one axis (e.g., z) the "layers" don't have to be evenly spaced. Furtunatly my data is almost in a grid shape which is fixable by interpolation.
Question: I have a function get_appendable_values(sequence) that takes a sequence (even empty) and returns a list of all the values appendable (as a last element) to that sequence. I need to generate all the possible sequences of 4 elements, with respect to the rules defined in this function and starting with an empty sequence. Example : Let's say the implementation of get_appendable_values is : def get_appendable_values(sequence): '''Dummy rules''' if len(sequence) == 2: return [4, 12] if sequence[-1] == 4: return [7] return [0, 9]  Expected output : [[0, 0, 4, 7], [0, 0, 12, 0], [0, 0, 12, 9], [0, 9, 4, 7], [0, 9, 12, 0], [0, 9, 12, 9], [9, 0, 4, 7], [9, 0, 12, 0], [9, 0, 12, 9], [9, 9, 4, 7], [9, 9, 12, 0], [9, 9, 12, 9]]  I have the feeling that recursion is the key, but I could not figure it out.

Answer: Yes, recursion is the key. To generate a sequence of size 4, you first generate all sequences of size 3, and add all possible endings to them. Likewise, to generate a sequence of size 3, you need all sequences of size 2... and so forth down to size 0. def get_appendable_values(sequence): '''Dummy rules''' if len(sequence) == 2: return [4, 12] #need a len check here to avoid IndexError when `sequence` is empty if len(sequence) > 0 and sequence[-1] == 4: return [7] return [0, 9] def generate_sequences(size): if size == 0: yield [] else: for left_part in generate_sequences(size-1): for right_part in get_appendable_values(left_part): yield left_part + [right_part] for seq in generate_sequences(4): print(seq)  Result: [0, 0, 4, 7] [0, 0, 12, 0] [0, 0, 12, 9] [0, 9, 4, 7] [0, 9, 12, 0] [0, 9, 12, 9] [9, 0, 4, 7] [9, 0, 12, 0] [9, 0, 12, 9] [9, 9, 4, 7] [9, 9, 12, 0] [9, 9, 12, 9]
Question: I have the following pandas DataFrame in python:     attr_x header example_x other     3232 322 abv ideo   342 123213 ffee iie   232 873213 ffue iie   333 4534 ffpo iieu     I want to remove the suffixes '_x' from all columns containing it. The original DataFrame is much longer. Example result:     attr header example other     3232 322 abv ideo   342 123213 ffee iie   232 873213 ffue iie   333 4534 ffpo iieu    

Answer: Use str.removesuffix: df.columns = df.columns.str.removesuffix("_x")  Or replace: df.columns = df.columns.str.replace(r'_x$', '')
Question: Loading .gitlab-ci.yml fails when I try to load it with yaml.load. yaml.constructor.ConstructorError: could not determine a constructor for the tag '!reference' in "python.yml", line 9, column 7  That's the yaml I try to load. .python: before_script: - pip install -r requirements.txt script: - echo "Hello Python!" test: before_script: - !reference [.python, before_script] script: - pytest  Right now I'm not interested in those references. However, I'm modifying some parts of the yaml and then write it back to the filesystem. So, I don't want to strip these references.  Are there any GitLab libraries to load gitlab-ci.yml? I could not find any. Is there a way to treat !reference [.python, before_script] as String and keep it as is? 

Answer: Upgrade to ruamel.yaml>=0.15.x  yaml_str = """ .python: before_script: - pip install -r requirements.txt script: - echo "Hello Python!" test: before_script: - !reference [.python, before_script] script: - pytest """ from ruamel.yaml import YAML yaml = YAML() yaml.preserve_quotes = True data = yaml.load(yaml_str)
Question: I'm trying to run RegNeRF, which requires flax. On installing the latest version of flax==0.6.0, I got an error stating flax has no attribute optim. This answer suggested to downgrade flax to 0.5.1. On doing that, now I'm getting the error AttributeError: module 'flax' has no attribute 'nn' I could not find any solutions on the web for this error. Any help is appreciated. I'm using ubuntu 20.04

Answer: The flax.optim module has been moved to optax as of flax version 0.6.0; see Upgrading my Codebase to Optax for information on how to migrate your code. If you're using external code that imports flax.optim and can't update these references, you'll have to install flax version 0.5.3 or older. Regarding flax.nn: this module was replaced by flax.linen in flax version 0.4.0. See Upgrading my Codebase to Linen for information on this migration. If you're using external code that imports flax.nn and can't update these references, you'll have to install flax version 0.3.6 or older.
Question: Can someone help me turn this base64 data into image? I don't know if it's because the data was not decoded properly or anything else. Here is how I decoded the data: import base64 c_data = { the data in the link (string type) } c_decoded = base64.b64decode(c_data)  But it gave the error Incorrect Padding so I followed some tutorials and tried different ways to decode the data. c_decoded = base64.b64decode(c_data + '=' * (-len(c_data) % 4)) c_decoded = base64.b64decode(c_data + '=' * ((4 - len(c_data) % 4) % 4)  Both ways decoded the data without giving the error Incorrect Padding but now I can't turn the decoded data into image. I have tried creating an empty png then write the decoded data into it: from PIL import Image with open('c.png', 'wb') as f: f.write(c_decoded) image = Image.open('c.png') image.show()  It didn't work and gave the error: cannot identify image file 'c.png' I have tried using BytesIO: from PIL import Image import io from io import BytesIO image = Image.open(io.BytesIO(c_decoded)) image.show()  Now it gave the error: cannot identify image file <_io.BytesIO object at 0x0000024082B20270> Please help me.

Answer: Not sure if you definitely need a Python solution, or you just want help decoding your image like the first line of your question says, and you thought Python might be needed. If the latter, you can just use ImageMagick in the Terminal: cat YOURFILE.TXT | magick inline:- result.png   Or equivalently and avoiding "Useless Use of cat": magick inline:- result.png < YOURFILE.TXT   If the former, you can use something like this (untested): from urllib import request with request.urlopen('data:image/png;base64,iVBORw0...') as response: im = response.read()  Now im contains a PNG-encoded [^1] image, so you can either save to disk as such: with open('result.png','wb') as f: f.write(im)  Or, you can wrap it in a BytesIO and open into a PIL Image: from io import BytesIO from PIL import Image pilImage = Image.open(BytesIO(im))  [^1]: Note that I have blindly assumed it is a PNG, it may be JPEG, so you should ideally look at the start of the DataURI to determine a suitable extension for saving your file.
Question: I'm trying to make a scrapy scraper work using cloud run. The main idea is that every 20 minutes a cloud scheduler cron should trigger the web scraper and get data from different sites. All sites have the same structure, so I would like to use same code and parallelize the execution of the scraping job, doing something like scrapy crawl scraper -a site=www.site1.com and scrapy crawl scraper -a site=www.site2.com. I have already deployed a version of the scraper, but it only can do scrapy crawl scraper. How can I do that at execution the command's site change? Also, should I be using cloud run job or service?

Answer: According to that page of documentation, there is a trick.  Define a number of task, let's say, you set the number of task equal to the number of site to scrap. use the --task parameter for that In your container (or in Cloud Storage, but if you do that, you have to download the file before starting the process), add a file with 1 website to scrap per line. At runtime, use the CLOUD_RUN_TASK_INDEX environment variable. That variable indicate the number of the task in the execution. For each different number, pick a line in your file of websites (the number of the line equal to the env var value).  Like that, you can leverage Cloud Run jobs and parallelism.  The main tradeoff here is the static form of the websites list to scrap.
Question: I'm trying to plot a simple function using python, numpy and matplotlib but when I execute the script it returns a ValueError described in the title. This is my code: """Geometrical interpretation: In Python, plot the function y = f(x) = x**3 − (1/x) and plot its tangent line at x = 1 and at x = 2.""" import numpy as np import matplotlib.pyplot as plt def plot(func): plt.figure(figsize=(12, 8)) x = np.linspace(-100, 100, 100) plt.plot(x, func, '-', color='pink') plt.show() plt.close() plot(lambda x: x ** 3 - (1 / x))  Please send this beginer some help :)

Answer: This was a good effort actually. You only needed to add the y variable using y=func(x). And then plt.plot(x,y)... so this works: import numpy as np import matplotlib.pyplot as plt def plot(func): plt.figure(figsize=(12, 8)) x = np.linspace(-100, 100, 100) y = func(x) plt.plot(x, y, '-', color='pink') plt.show() plt.close() plot(lambda x: x ** 3 - (1 / x))  result:
Question: Install a python version using: $ pyenv install 3.8.9 Installed Python-3.8.9 to /Users/robino/.pyenv/versions/3.8.9  List the python versions now available: $ pyenv versions * system 3.8.2 3.8.9  A week goes by and I forget where it is installed. Now suppose I want to get the executable path for 3.8.9 version. The following do not work: $ pyenv which 3.8.9 pyenv: 3.8.9: command not found $ pyenv which python 3.8.9 (gives path to system python) $ pyenv which python-3.8.9 pyenv: python-3.8.9: command not found $ pyenv which Python-3.8.9 pyenv: Python-3.8.9: command not found  A workaround I found was to set the python version, check, then set it back to system: $ pyenv local 3.8.9 $ pyenv which python /Users/robino/.pyenv/versions/3.8.9/bin/python $ pyenv local --unset  However this is a suboptimal solution as it required that no local is previous set. What is the correct command to print out the python executable path for a currently not used version, using pyenv?

Answer: By default, pyenv executable can be found at $(pyenv root)/versions/{VERSION}/bin/python. I am not aware of a command displaying all/any executables other than pyenv which python. If you'd like to get the path via commands though, another option would be to make a temporary subdirectory and set the local pyenv interpreter there: $ mkdir tmp; cd tmp $ pyenv local 3.8.9 $ pyenv which python /Users/robino/.pyenv/versions/3.8.9/bin/python $ cd ..; rm -r tmp  Since deeper directories take priority with local pyenv versions, a parent directory wouldn't interfere in this case. Yet another option would be to temporarily set the global pyenv version, as this does not have the requirement of no local pyenv version being set. I wouldn't like this though as I'd probably forget to set it back to its original value ;)
Question: ds = Dataset([[1, 1, 1, 2, 2, 2], ["foo", "bar", "monty", "foo", "bar", "monty"], ["a", "b", "c", "d", "e", "f"], [1, 2, 3, 4, 5, 6]], [:g, :key, :foo, :bar])  In InmemoryDatasets, the transpose function can Pass Tuple of column selectors. transpose(groupby(ds, :g), (:foo, :bar), id = :key)  Result: g foo bar monty foo_1 bar_1 monty_1 identity identity identity identity identity identity identity Int64? String? String? String? Int64? Int64? Int64? 1 1 a b c 1 2 3 2 2 d e f 4 5 6  Question: How can I do this in DataFrames.jl? How can I do this in R and Python?

Answer: In R, pivot_wider can be used for reshaping. library(tidyr) pivot_wider(ds, names_from = key, values_from = c(foo, bar))  -output # A tibble: 2 × 7 g foo_foo foo_bar foo_monty bar_foo bar_bar bar_monty <dbl> <chr> <chr> <chr> <int> <int> <int> 1 1 a b c 1 2 3 2 2 d e f 4 5 6   If we want to get the same column names, we could rename the columns library(dplyr) library(stringr) ds %>% rename("grp"= 'foo', '1' = 'bar') %>% pivot_wider(names_from = key, values_from = c("grp", `1`), names_glue = "{key}_{.value}") %>% rename_with(~ str_remove(.x, "_grp"), ends_with('_grp'))  -output # A tibble: 2 × 7 g foo bar monty foo_1 bar_1 monty_1 <dbl> <chr> <chr> <chr> <int> <int> <int> 1 1 a b c 1 2 3 2 2 d e f 4 5 6  data ds <- structure(list(g = c(1, 1, 1, 2, 2, 2), key = c("foo", "bar", "monty", "foo", "bar", "monty"), foo = c("a", "b", "c", "d", "e", "f"), bar = 1:6), class = "data.frame", row.names = c(NA, -6L))
Question: I have the following dataset: df = pd.DataFrame([ {'Phone': 'Fax(925) 482-1195', 'Fax': None}, {'Phone': 'Fax(406) 226-0317', 'Fax': None}, {'Phone': 'Fax+1 650-383-6305', 'Fax': None}, {'Phone': 'Phone(334) 585-1171', 'Fax': 'Fax(334) 585-1182'}, {'Phone': None, 'Fax': None}, {'Phone': 'Phone(334) 585-1171', 'Fax': 'Fax(334) 585-1176'}] )  Which should look like:  What I'm trying to do is: for every row that I see "Fax", I want to truncate it and transfer this record to the column "Fax". At first, I was trying to query only the matching with this filtering: df[df['Phone'].str.contains("Fax") == True, "Fax"] = df[df['Phone'].str.contains("Fax") == True]  But it does not works, with the error: "TypeError: unhashable type: 'Series'". Any ideas?

Answer: You have a bunch of rows, that is, a list of dicts. Simplest approach would be to massage each row prior to adding it to the dataframe. rows = [ ... ] def get_contacts(rows): for row in rows: phone, fax = row['Phone'], row['Fax'] if 'Fax' in phone: phone, fax = None, phone yield phone, fax df = pd.DataFrame(get_contacts(rows))   You can force str instead of None with a filter like this:  ... yield clean(phone), clean(fax) ... def clean(s, default=''): if s is None: return default return s   If you really prefer to stick to using Pandas, you might want to  identify a mask of rows where df.Phone contains 'Fax', then copy that subset into df['Fax'], then blank out selected df['Phone'] entries.  You can verify / debug each step by itself -- get (1) right before moving on to attempt (2). If you choose to go this route, please post your final solution.
Question: I have the following in my pyproject.toml [tool.pytest.ini_options] markers = [ "plot: marks SLOW plot tests (deselect with '-m \"not plot\"')", "open_tutorial: marks the open_tutorial (which opens VSCode all the times)" ]  and I have a bunch of test methods marked accordingly. If I run coverage run --branch -m pytest -m "not open_tutorial"  or coverage run --branch -m pytest -m "not plot"  I got the desired results, namely the marked test are skipped, but I cannot figure out how to make pytest to skip both. I tried the following coverage run --branch -m pytest -m "not open_tutorial" -m "not plot" coverage run --branch -m pytest -m "not open_tutorial" "not plot" coverage run --branch -m pytest -m ["not open_tutorial","not plot"]  but none of them worked.

Answer: According to pytest help:  -m MARKEXPR only run tests matching given mark expression. For example: -m 'mark1 and not mark2'.  If you want to use more than one marker you should use the and operator. pytest -m "not open_tutorial and not plot"  will run all test without marks: open_tutorial and plot but: pytest -m "not open_tutorial and not plot and othermark"  will run tests with othermark if they don't have plot or open_tutorial marks.
Question: For an array of multiple dimensions, I would like to sum along some dimensions, with the sum range defined by other dimension indexes. Here is an example: >>> import numpy as np >>> x = np.arange(2*3*4).reshape((2,3,4)) >>> x array([[[ 0, 1, 2, 3], [ 4, 5, 6, 7], [ 8, 9, 10, 11]], [[12, 13, 14, 15], [16, 17, 18, 19], [20, 21, 22, 23]]]) >>> wanted = [[sum(x[i,j,i:j]) for j in range(x.shape[1])] for i in range(x.shape[0])] >>> wanted [[0, 4, 17], [0, 0, 21]]  Is there a more efficient way to do it without for loops or list comprehension? My array is quite large.

Answer: You can use boolean masks: # get lower triangles m1 = np.arange(x.shape[1])[:,None]>np.arange(x.shape[2]) # get columns index >= depth index m2 = np.arange(x.shape[2])>=np.arange(x.shape[0])[:,None,None] # combine both mask to form 3D mask mask = m1 & m2 out = np.where(mask, x, 0).sum(axis=2)  output: array([[ 0, 4, 17], [ 0, 0, 21]])  Masks: # m1 array([[False, False, False, False], [ True, False, False, False], [ True, True, False, False]]) # m2 array([[[ True, True, True, True]], [[False, True, True, True]]]) # mask array([[[False, False, False, False], [ True, False, False, False], [ True, True, False, False]], [[False, False, False, False], [False, False, False, False], [False, True, False, False]]])
Question: Te following code is used to connect with the opcua server: from opcua import Client client = Client(url='opc.tcp://192.168.0.5:4840') client.set_user('user1') client.set_password('password') client.connect()  Error message: Received an error: MessageAbort(error:StatusCode(BadSecurityPolicyRejected), reason:None) Protocol Error I also tried to append this code: client.set_security_string("Basic256Sha256,Sign,cert.pem,key.pem")  But I do not know where I can create de cert.pem and key.pem Does anyone know how to connect with the server in python

Answer: You can use this script and adapt it to your needs: https://github.com/FreeOpcUa/python-opcua/blob/master/examples/generate_certificate.sh Also python-opcua is deprecated. So if you start a new project, I would recommend you to use asyncua. Asyncua also has a sync wrapper if you don't want to deal with asyncio.
Question: I have a dataframe (sample) like this: import pandas as pd data = [['A', '2022-09-01', False, 2], ['A', '2022-09-02', False, 3], ['A', '2022-09-03', True, 1], ['A', '2022-09-05', False, 4], ['A', '2022-09-08', True, 4], ['A', '2022-09-09', False, 2], ['B', '2022-09-03', False, 4], ['B', '2022-09-05', True, 5], ['B', '2022-09-06', False, 7], ['B', '2022-09-09', True, 4], ['B', '2022-09-10', False, 2], ['B', '2022-09-11', False, 3]] df = pd.DataFrame(data = data, columns = ['group', 'date', 'indicator', 'val']) group date indicator val 0 A 2022-09-01 False 2 1 A 2022-09-02 False 3 2 A 2022-09-03 True 1 3 A 2022-09-05 False 4 4 A 2022-09-08 True 4 5 A 2022-09-09 False 2 6 B 2022-09-03 False 4 7 B 2022-09-05 True 5 8 B 2022-09-06 False 7 9 B 2022-09-09 True 4 10 B 2022-09-10 False 2 11 B 2022-09-11 False 3  I would like to create a column called Diff, which shows the difference of rows with its nearest (depends on date) conditional rows (indicator == True) where the conditional rows have a value of 0 per group. Here is the desired output: data = [['A', '2022-09-01', False, 2, 1], ['A', '2022-09-02', False, 3, 2], ['A', '2022-09-03', True, 1, 0], ['A', '2022-09-05', False, 4, 3], ['A', '2022-09-08', True, 4, 0], ['A', '2022-09-09', False, 2, -2], ['B', '2022-09-03', False, 4, -1], ['B', '2022-09-05', True, 5, 0], ['B', '2022-09-06', False, 7, 2], ['B', '2022-09-09', True, 4, 0], ['B', '2022-09-10', False, 2, -2], ['B', '2022-09-11', False, 3, -1]] df_desired = pd.DataFrame(data = data, columns = ['group', 'date', 'indicator', 'val', 'Diff']) group date indicator val Diff 0 A 2022-09-01 False 2 1 1 A 2022-09-02 False 3 2 2 A 2022-09-03 True 1 0 3 A 2022-09-05 False 4 3 4 A 2022-09-08 True 4 0 5 A 2022-09-09 False 2 -2 6 B 2022-09-03 False 4 -1 7 B 2022-09-05 True 5 0 8 B 2022-09-06 False 7 2 9 B 2022-09-09 True 4 0 10 B 2022-09-10 False 2 -2 11 B 2022-09-11 False 3 -1  As you can see it returns the difference respectively with the nearest indicator == True rows per group where the conditioned rows have a Diff of 0. So I was wondering if anyone knows have to get the desired result using pandas? Extra info column Diff: Let's take group A as an example. The column Diff is calculated by the difference with respect to the nearest row with indicator True. So for example:  Row 1 is 2 - 1 = 1 with respect to row 3 (nearest True row based on date). Row 2 is 3 - 1 = 2 with respect to row 3. Row 4 is 4 - 1 = 3 with respect to row 3. Row 6 is 2 - 4 = -2 with respect to row 5 (nearest True row based on date). The rows with True have a value of 0 in Diff because everything is calculated with respect to these rows. 

Answer: IIUC use merge_asof with filtered rows by indicator and subtract column val: df['date'] = pd.to_datetime(df['date'] ) df = df.sort_values('date') df['Diff'] = df['val'].sub(pd.merge_asof(df, df[df['indicator']], on='date', by='group', direction='nearest')['val_y']) df = df.sort_index()