eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: Given a df as follow  time_interval dvalue 0 (0, 5] 1 1 (5, 10] 2 2 (10, 15] 4 3 (15, 20] 5 4 (20, 25] 6 5 (25, 30] 7 6 (30, 35] 8  I would like to split the column time_interval, which of type interval[int64, right] as the following  dvalue l u 0 1 0 5 1 2 5 10 2 4 10 15 3 5 15 20 4 6 20 25 5 7 25 30 6 8 30 35  The full code to reproduce the df is as follow ls = range(0, 100, 5) df=pd.DataFrame([1,2,4,5,6,7,8],columns=['dvalue']) df.index = pd.IntervalIndex.from_breaks(ls[:len(df) + 1]) df.reset_index(inplace=True) df.rename(columns={'index':'time_interval'},inplace=True) 

Answer: Use Interval.left and Interval.right: df['l'] = df['time_interval'].apply(lambda x: x.left) df['u'] = df['time_interval'].apply(lambda x: x.right)   df['l'] = df['time_interval'].map(lambda x: x.left) df['u'] = df['time_interval'].map(lambda x: x.right)  Or first convert to IntervalIndex: idx = pd.IntervalIndex(df['time_interval']) df['l'] = idx.left df['u'] = idx.right   idx = pd.IntervalIndex(df['time_interval']) df = df.assign(l=idx.left, u=idx.right) print (df) time_interval dvalue l u 0 (0, 5] 1 0 5 1 (5, 10] 2 5 10 2 (10, 15] 4 10 15 3 (15, 20] 5 15 20 4 (20, 25] 6 20 25 5 (25, 30] 7 25 30 6 (30, 35] 8 30 35
Question: I'm seeking to scrape a web page using Playwright. I load the page, and click the download button with Playwright successfully. This brings up a print dialog box with a printer selected.  I would like to select "Save as PDF" and then click the "Save" button. Here's my current code: with sync_playwright() as p: browser = p.chromium.launch(headless=True) playwright_page = browser.new_page() got_error = False try: playwright_page.goto(url_to_start_from) print(playwright_page.title()) html = playwright_page.content() except Exception as e: print(f"Playwright exception: {e}") got_error = True if not got_error: soup = BeautifulSoup(html, 'html.parser') #download pdf with playwright_page.expect_download() as download_info: playwright_page.locator("text=download").click() download = download_info.value path = download.path() download.save_as(DOWNLOADED_PDF_FOLDER) browser.close()  Is there a way to do this using Playwright?

Answer: Thanks very much to @KJ in the comments, who suggested that with headless=True, Chromium won't even put up a print dialog box in the first place.
Question: I am trying to pickle a nested dictionary which is created using: collections.defaultdict(lambda: collections.defaultdict(int))  My simplified code goes like this: class A: def funA(self): #create a dictionary and fill with values dictionary = collections.defaultdict(lambda: collections.defaultdict(int)) ... #then pickle to save it pickle.dump(dictionary, f)  However it gives error: AttributeError: Can't pickle local object 'A.funA.<locals>.<lambda>'  After I print dictionary it shows: defaultdict(<function A.funA.<locals>.<lambda> at 0x7fd569dd07b8> {...}  I try to make the dictionary global within that function but the error is the same. I appreciate any solution or insight to this problem. Thanks!

Answer: pickle records references to functions (module and function name), not the functions themselves. When unpickling, it will load the module and get the function by name. lambda creates anonymous function objects that don't have names and can't be found by the loader. The solution is to switch to a named function. def create_int_defaultdict(): return collections.defaultdict(int) class A: def funA(self): #create a dictionary and fill with values dictionary = collections.defaultdict(create_int_defaultdict) ... #then pickle to save it pickle.dump(dictionary, f)
Question: I have a webservice that give doc list. I call this webservice via get_doc_list. but when I pass 2 values to id__in, it return one mapping object. def get_doc_list(self, id__in): config = self.configurer.doc params = { "id__in": id__in, } response = self._make_request( token=self.access_token, method='get', proxies=self.proxies, url=config.service_url, params=params, module_name=self.module_name, finalize_response=False ) return response  How can I fix it?!

Answer: You can add this two lines before make_request: string_id_in = [str(i) for i in id_in] id_in = ",".join(string_id_in)
Question: Im new here. I have problem with this code, #Library import numpy as np import pickle import cv2 from os import listdir from sklearn.preprocessing import LabelBinarizer from keras.models import Sequential from keras.layers import BatchNormalization from keras.layers.convolutional import Conv2D from keras.layers.convolutional import MaxPooling2D from keras.layers.core import Activation, Flatten, Dropout, Dense from keras import backend as K from keras.preprocessing.image import ImageDataGenerator from keras.optimizers import Adam from keras.preprocessing import image #from tensorflow.keras.preprocessing.image import img_to_array from keras.preprocessing.image import img_to_array from sklearn.preprocessing import MultiLabelBinarizer from sklearn.model_selection import train_test_split import matplotlib.pyplot as plt  I got an error  this code is from github link Im using  python 3.7.13 tensorflow 2.9 opencv 4.5.5 keras 2.9.0 

Answer: In Keras Documentation V2.9.0, In tf version 2.9.0 the img_to_array moved to utlis Insted of, from keras.preprocessing.image import img_to_array Try this, from tensorflow.keras.utils import img_to_array
Question: So here's my simple example (the json field in my actual dataset is very nested so I'm unpacking things one level at a time). I need to keep certain columns on the dataset post json_normalize(). https://pandas.pydata.org/docs/reference/api/pandas.json_normalize.html Start:  Expected (Excel mockup):  Actual:  import json d = {'report_id': [100, 101, 102], 'start_date': ["2021-03-12", "2021-04-22", "2021-05-02"], 'report_json': ['{"name":"John", "age":30, "disease":"A-Pox"}', '{"name":"Mary", "age":22, "disease":"B-Pox"}', '{"name":"Karen", "age":42, "disease":"C-Pox"}']} df = pd.DataFrame(data=d) display(df) df = pd.json_normalize(df['report_json'].apply(json.loads), max_level=0, meta=['report_id', 'start_date']) display(df)  Looking at the documentation on json_normalize(), I think the meta parameter is what I need to keep the report_id and start_date but it doesn't seem to be working as the expected fields to keep are not appearing on the final dataset. Does anyone have advice? Thank you.

Answer: as you're dealing with a pretty simple json along a structured index you can just normalize your frame then make use of .join to join along your axis. from ast import literal_eval df.join( pd.json_normalize(df['report_json'].map(literal_eval)) ).drop('report_json',axis=1) report_id start_date name age disease 0 100 2021-03-12 John 30 A-Pox 1 101 2021-04-22 Mary 22 B-Pox 2 102 2021-05-02 Karen 42 C-Pox
Question: When I run the following code to determine my device's local IP address, I get 127.0.0.1 instead of 192.168.0.101. import socket import threading PORT = 8080 HOST_NAME = socket.gethostname() print(HOST_NAME) SERVER = socket.gethostbyname(HOST_NAME) print(SERVER)  The output i get on the console is MyDeviceName.local 127.0.0.1 

Answer: 127.0.0.1 is localhost address, it is right. If you want your device's address do this: import socket s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM) s.connect(("8.8.8.8", 80)) print(s.getsockname()[0])
Question:  OS: win10 PyCharm version: PyCharm Professional 2021.2.2 Poetry version: 1.1.13 Poetry plugin version: 1.1.5-212 (from koudai aono)  I have tried to build a new PyCharm project by poetry environment,  while setting up it showed  and cannot setup the interpreter. Have anyone got the similar problem before and know how to solve this error? Update I have upgraded the PyCharm version to PyCharm Professional 2022.1.1 and the problem still remain.

Answer: Alright, I have fixed this problem. Below is my debug steps, hope it can help those who are struggling in the same situation:  Posting a support report to PyCharm team with no responce Searching for a lot of posts from communities Creating a poetry project by terminal Inside the project directory, I tried poetry env info and it showed that the local virtualenv is NA Trying to create one by poetry env use $(which python), yet it returned Skipping virtualenv creation, as specified in config file and this answer gave me a hint. Typing poetry config --list and it showed that virtualenvs.create = false Trying to enable the virtualenv creation by the command poetry config virtualenvs.create true Restarted the PyCharm IDE and tried to add the interpreter again and it WORKED!!  I am not sure if the command poetry config virtualenvs.create true is permant or not.
Question: This answer does not seem to work for generics. Mypy complains about "error: Missing type parameters for generic type A" when checking the following code. I have tried using 'A[T]' for the TypeVar but then mypy says "error: Type variable T is unbound." I have also tried using AnyA[T] as return type of get but that produces two error messages, the already known "error: Missing type parameters for generic type A" and the new error message "Type variable AnyA used with arguments". How do I specify the return type of get correctly? import typing T = typing.TypeVar('T') AnyA = typing.TypeVar('AnyA', bound='A') class A(typing.Generic[T]): def __init__(self, val: T) -> None: self.val = val def get(self: AnyA) -> AnyA: return self class B(A[T]): def is_int(self) -> bool: return isinstance(self.val, int) if __name__ == '__main__': b = B(42) print(b.get().is_int()) 

Answer: I know of three ways of typing here: Declaring an inner self-type This approach is described in mypy docs, see Precise typing of alternative constructors. class A(typing.Generic[T]): _Self = typing.TypeVar('_Self', bound='A[T]') def __init__(self, val: T) -> None: self.val = val def get(self: _Self) -> _Self: return self  Note however, that this is mypy-specific stuff and may not work with other checkers. E.g. pyre doesn't support inner self-types yet. Using _typeshed.Self This saves the boilerplate of declaring custom types, but requires a somewhat obscure import from typeshed which will fail at runtime. It thus must be wrapped by typing.TYPE_CHECKING: from typing import Any, TYPE_CHECKING if TYPE_CHECKING: from _typeshed import Self else: Self = Any class A(typing.Generic[T]): def __init__(self, val: T) -> None: self.val = val def get(self: Self) -> Self: return self  _typeshed.Self was created to be used in custom stubs in the first place, but is suitable for inline typing as well. Python 3.11 and upwards: typing.Self A recently introduced PEP 673 adds Self to stdlib, so starting from Python 3.11 one will be able to use that: from typing import Self class A(typing.Generic[T]): def __init__(self, val: T) -> None: self.val = val def get(self: Self) -> Self: return self  This is not supported by mypy yet as of now though, but e.g. by pyright from version 1.1.184.
Question: I have around 12 million records that I need to update in my postgres db (so I need to do it in an efficient way). I am using Django. I have to update a jsonfield column (extra_info) to use values from a different column (related_type_id which is an integer) in the same model. Trying to do it with an update. This seems to be the way to do it most efficiently. Example: Person.objects.all().update(extra_info={ "type": "Human", "id": F('related_type_id') })  This errors out with : "Object of type F is not JSON serializable". I thought the F() will give back the value of that column (which should be an integer) which should be serializable. Can this be done ? Or am I trying to do it in the wrong way. I don't really want to iterate in a for loop and update each record with a save() because it will take too long. There's too many records.

Answer: The Django database function JSONObject should work, it returns a valid JSON object from key-value pairs and may be passed F objects from django.db.models import F, Value from django.db.models.functions import JSONObject Person.objects.all().update(extra_info=JSONObject( type=Value('Human'), id=F('related_type_id') ))
Question: I'm on day 1 of Python and trying to import SciPy into a project. I installed it via pip install on ElementaryOS (an Ubuntu derivative). I have verified it's existence via: $ python >>> help("modules")  The exact error I'm getting is:  Import "scipy" could not be resolved Pylance (reportMissingImports)  When searching for this error I found:  Import could not be resolved/could not be resolved from source Pylance in VS Code using Python 3.9.2 on Windows 10 Powershell -- the accepted answers all pointed towards a project specific .env file. I have no such project structure, nor does it make sense to me that one would be needed.  A github issue -- this issue ends with "it just fixed itself"   When I run my program, I get no errors in console. And looking up "Pylance" it appears to be a Microsoft product. I suspect that VSCode is failing to lint correctly. Potentially because pip installed something in a place it wasn't expecting. This is my guess, but any help would be very much appreciated.  Edit: Following through on the idea of missing paths, I found this post -- How do I get into the environment VS Code is using for pylance? Having added the path to where my modules can be found has yielded no results, though I'm not sure if the formatting is correct. Perhaps it needs glob syntax (eg path/**/*) 

Answer: The issue was indeed with Pylance. It was missing an "additional path" to where pip had installed the projects I wanted to import. To solve the issue: First make sure you know the location of your import; you can find it with: $ python >>> import modulename >>> print(modulename.__file__)  Then, once you know the location:  Open settings (ctrl + ,) Search "pylance" or find it under "Extensions > Pylance" Find the "Extra Paths" config item Use "add item" to a add a path to the parent folder of the module. It will not do any recursive tree searching  And you should be good to go! For a further example, you can see the image above where I had added the path /home/seph/.local/lib/python2.7/ to no avail. Updating it to /home/seph/.local/lib/python2.7/site-packages/ did the trick.
Question: While experimenting with Numpy, I found that the contiguous value provided by numpy.info may differ from numpy.ndarray.data.contiguous (see the code and screenshot below). import numpy as np x = np.arange(9).reshape(3,3)[:,(0,1)] np.info(x) print(f''' {x.data.contiguous = } {x.flags.contiguous = } {x.data.c_contiguous = } {x.flags.c_contiguous = } {x.data.f_contiguous = } {x.flags.f_contiguous = } ''')  According to documentation about a memoryview class, data.contiguous == True exactly if an array is either C-contiguous or Fortran contiguous. As for numpy.info, I believe it displays the value of flags.contiguous. Alas, there is no information about it in the manual. What does it actually mean? Is it a synonim for flags.c_contiguous? 

Answer: In the source code of numpy.info, we can see the subroutine for processing ndarray: def info(object=None, maxwidth=76, output=None, toplevel='numpy'): ... elif isinstance(object, ndarray): _info(object, output=output) ... def _info(obj, output=None): """Provide information about ndarray obj""" bp = lambda x: x ... print("contiguous: ", bp(obj.flags.contiguous), file=output) print("fortran: ", obj.flags.fortran, file=output) ...  It returns flags.contiguous as the array's continuity parameter. This one isn't specified in flags description. But we can find it in flagsobject.c: // ... static PyGetSetDef arrayflags_getsets[] = { {"contiguous", (getter)arrayflags_contiguous_get, NULL, NULL, NULL}, {"c_contiguous", (getter)arrayflags_contiguous_get, NULL, NULL, NULL}, // ...  So it's clear now that a contiguous parameter from numpy.info is actually flags.c_contiguous and has nothing in common with ndarray.data.contiguous. I guess when programming in C it was natural to say just contiguous instead of c_contiguous, and this has led to a slight inconsistency in terminology.
Question: I have two data frames: main and auxiliary. I am concatenating auxiliary to the main. It results in NaN in a few rows and I want to fill them, not all. Code: df1 = pd.DataFrame({'Main':[00,10,20,30,40,50,60,70,80]}) df1 = Main 0 0 1 10 2 20 3 30 4 40 5 50 6 60 7 70 8 80 df2 = pd.DataFrame({'aux':['aa','aa','bb','bb']},index=[0,2,5,7]) df2 = aux 0 aa 2 aa 5 bb 7 bb df = pd.concat([df1,df2],axis=1) # After concating, in the aux column, I want to fill the NaN rows in between # the rows with same value. Example, fill rows between 0 and 2 with 'aa', 2 and 5 NaN, 5 and 7 with 'bb' df = pd.concat([df1,df2],axis=1).fillna(method='ffill') print(df)  Present result:  Main aux 0 0 aa 1 10 aa 2 20 aa 3 30 aa # Wrong, here it should be NaN 4 40 aa # Wrong, here it should be NaN 5 50 bb 6 60 bb 7 70 bb 8 80 bb # Wrong, here it should be NaN  Expected result:  Main aux 0 0 aa 1 10 aa 2 20 aa 3 30 NaN 4 40 NaN 5 50 bb 6 60 bb 7 70 bb 8 80 NaN 

Answer: If I understand correctly, what you want can be done like this. You want to fill the NaNs where backfill and forward fill give the same value. ff = df.aux.ffill() bf = df.aux.bfill() df.aux = ff[ff == bf]
Question: A colleague and I just stumbled across an interesting problem using an f-string. Here is a minimal example: >>> f"{ 42:x}" '2a'  Writing a space after the hexadecimal type leads to a ValueError: >>> f"{ 42:x }" Traceback (most recent call last): File "<stdin>", line 1, in <module> ValueError: Invalid format specifier  I understand the paragraph Leading and trailing whitespace in expressions is ignored in PEP 498 to mean that the space should actually be ignored. Why does the space character lead to an error and what is the reasoning behind this?

Answer: Per the link you shared:  For ease of readability, leading and trailing whitespace in expressions is ignored. This is a by-product of enclosing the expression in parentheses before evaluation.  The expression is everything[1] before the colon (:), while the format specifier is everything afterwards. { 42 : x } inside an f-string is equivalent to ( 42 ) with a format specifier of " x ". ( 42 ) == 42 but " x " != "x", and a format specifier is passed to the object's __format__ intact. When the object attempts to format, it doesn't recognize the meaning of space (" ") in those positions, and throws an error. Furthermore, on some format specifiers, space actually has a meaning, for example as a fill character or as a sign option: >>> f"{42: >+5}" # Width of 5, align to right, with space as fill and plus or minus as sign ' +42' >>> f"{42:#> 5}" # Width of 5, align to right, with `#` as fill and space or minus as sign. '## 42'  For more info, see the Format Specification Mini-Language.  1 Does not include the type conversion, for example !s, !r or !a.
Question: I have a string format that can be changed by someone else (just say) sample = f"This is a {pet} it has {number} legs"  And I have currently two string a = "This is a dog it has 4 legs" b = "This was a dog"  How to check which string satisfies this sample format? I can use python's string replace() on sample and create regex of it and check using re.match. But the catch is sample can be changed, so statically using replace won't always work, as sample may get more place holders.

Answer: I liked the approaches but I found a two liner solution: (I don't know the performance aspect of this, but it works!)  def pattern_match(input, pattern): regex = re.sub(r'{[^{]*}','(.*)', "^" + pattern + "$") if re.match(regex, input): print(f"'{input}' matches the pattern '{pattern}'") pattern_match(a, sample) pattern_match(b, sample)  Output 'This is a dog it has 4 legs' matches the pattern 'This is a {pet} it has {number} legs'
Question: I have an extremely basic script that pops up a message with a button with the command ?place Upon clicking this button the bot replies Hi to the user who clicked it. If the button isn't interacted with for > approx 3 minutes it then starts to return "interaction failed".  after that the button becomes useless. I assume there is some sort of internal timeout i can't find in the docs. The button does the same thing whether using discord.py (2.0) or pycord. Nothing hits the console. It's as if the button click isn't picked up. Very occasionally the button starts to work again and a host of these errors hit the console: discord.errors.NotFound: 404 Not Found (error code: 10062): Unknown interaction Ignoring exception in view <View timeout=180.0 children=1> for item <Button style=<ButtonStyle.success: 3> url=None disabled=False label='click me' emoji=None row=None>:  I assume the timeout = 180 is the cause of this issue but is anyone aware of how to stop this timeout and why it's happening? I can't see anything in the docs about discord buttons only being usable for 3 mins. import discord from discord.ext import commands intents = discord.Intents.default() intents.members = True intents.message_content = True bot = commands.Bot(command_prefix="?", intents=intents) embed1=discord.Embed(title="Test", description = f"TESTING",color=0xffffff) print("bot connected") @ bot.command(name='place') async def hello(ctx): view = discord.ui.View() buttonSign = discord.ui.Button(label = "click me", style= discord.ButtonStyle.green) async def buttonSign_callback(interaction): userName = interaction.user.id embedText = f"test test test" embed=discord.Embed(title="Test", description = embedText,color=0xffffff) await interaction.response.send_message(f"Hi <@{userName}>") buttonSign.callback = buttonSign_callback view.add_item(item=buttonSign) await ctx.send(embed = embed1,view = view) bot.run(TOKEN) 

Answer: Explanation By default, Views in discord.py 2.0 have a timeout of 180 seconds (3 minutes). You can fix this error by passing in None as the timeout when creating the view. Code @bot.command(name='place') async def hello(ctx): view = discord.ui.View(timeout=None)  References discord.ui.View.timeout
Question: I am trying to process images and I would like to brighten the dark areas of my image. I tried Histogram Equalization, however as there are also some bright areas in the image, the result is not satisfying. This is why I am looking for a way to brighten only the dark areas of the image. As an example, Input image is on the left and the expected result is on the right, the hair and face of the girl are brightened   ImageMagick seems to offer some possibilities to achieve this, however, I would like to do it using python

Answer: If you want to avoid colour distortions, you could:  convert to HSV colourspace, split the channels, bump up the V (Value) channel recombine the channels save  That might go something like this: from PIL import Image # Open the image im = Image.open('hEHxh.jpg') # Convert to HSV colourspace and split channels for ease of separate processing H, S, V = im.convert('HSV').split() # Increase the brightness, or Value channel # Change 30 to 50 for bigger effect, or 10 for smaller effect newV = V.point(lambda i: i + int(30*(255-i)/255)) # Recombine channels and convert back to RGB res = Image.merge(mode="HSV", bands=(H,S,newV)).convert('RGB') res.save('result.jpg')   Essentially, I am changing the brightness from the black mapping to the green mapping:   Reminder to forgetful self... "you made the plot like this, Mark": import matplotlib.pyplot as plt import numpy as np # Generate some straight-line data xdata = np.arange(0,256) # And the new mapping ydata = xdata + 30*(255-xdata)/255 # Plot plt.plot(xdata,xdata,'.k') plt.plot(xdata,ydata,'g^') plt.title('Adjustment of V') plt.xlabel('Input V') plt.ylabel('Output V') plt.grid(True) plt.show()