eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I have custom inheriting exceptions in my Python project and I want to deprecate one of them. What is the proper way of doing it? Exceptions I have: class SDKException(Exception): pass class ChildException(SDKException): pass class ChildChildException(ChildException): # this one is to be deprecated pass  I want to deprecate the ChildChildException, considering the exception is used, raised and chained with other exceptions in the project.

Answer: You could use a decorator which shows a warning DeprecationWarning category on each instantiation of exception class: import warnings warnings.filterwarnings("default", category=DeprecationWarning) def deprecated(cls): original_init = cls.__init__ def __init__(self, *args, **kwargs): warnings.warn(f"{cls.__name__} is deprecated", DeprecationWarning, stacklevel=2) original_init(self, *args, **kwargs) cls.__init__ = __init__ return cls class SDKException(Exception): pass class ChildException(SDKException): pass @deprecated class ChildChildException(ChildException): # this one is to be deprecated pass try: raise ChildChildException() except ChildChildException: pass  app.py:7: DeprecationWarning: ChildChildException is deprecated  Update: Also, you can create custom warning class and pass it to the warn function: class ExceptionDeprecationWarning(Warning): pass warnings.warn(f"{cls.__name__} is deprecated", ExceptionDeprecationWarning)
Question: I have a conda environment with python=3.6.0 and all it's dependencies. Now I would like to use this yaml file to create another environment with the same dependencies but with python=3.7.0 without the need for installing the packages with right version one by one.

Answer: # Activate old environment conda activate so # Save the list of package to a file: conda list > log # Extract the package name but not the version or hash cat log | awk '{print $1}' > log2 # make the list of packages tr '\n' ' ' < log2 > log3 # print the list of packages cat log3  Use notepad to replace python by python==3.7. Then create a new environment with the edited list of packages. conda create --name so2 conda activate so2 conda install _libgcc_mutex ... python==3.7 ... zstd  Conda will try to install all packages with the same name but different version.
Question: I managed to run Pyodide in browser. I created hello.txt file. But how can I access it. Pyodide  https://github.com/iodide-project/pyodide/blob/master/docs/using_pyodide_from_javascript.md pyodide.runPython('open("hello.txt", "w")')  What I tried in chrome devtools? pyodide.runPython('os.chdir("../")') pyodide.runPython('os.listdir()') pyodide.runPython('os.path.realpath("hello.txt")')  Output for listdir ["hello.txt", "lib", "proc", "dev", "home", "tmp"]  Output for realpath "/hello.txt"  Also, pyodide.runPython('import platform') pyodide.runPython('platform.platform()')  Output "Emscripten-1.0-x86-JS-32bit"  All outputs in chrome devtools console. It is created in root folder. But how it can be accessed in file explorer or anyway to copy file to Download folder? Thanks

Answer: Indeed pyodide operates in an in-memory (MEMFS) filesystem created by Emscripten. You can't directly write files to disk from pyodide since it's executed in the browser sandbox. You can however, pass your file to JavaScript, create a Blob out of it and then download it. For instance, using, let txt = pyodide.runPython(` with open('/test.txt', 'rt') as fh: txt = fh.read() txt `); const blob = new Blob([txt], {type : 'application/text'}); let url = window.URL.createObjectURL(blob); window.location.assign(url);  It should have been also possible to do all of this from the Python side, using the type conversions included in pyodide, i.e. from js import Blob, document from js import window with open('/test.txt', 'rt') as fh: txt = fh.read() blob = Blob.new([txt], {type : 'application/text'}) url = window.URL.createObjectURL(blob) window.location.assign(url)  however at present, this unfortunately doesn't work, as it depends on pyodide#788 being resolved first.
Question: I have a data frame that contains duplicates. and I would like to remove these duplicates. I also found this function from pandas df.drop_duplicates(subset=['Action', 'Name']). Unfortunately, this function removes too much, because only if the time is less than or equal to 5 minutes should it be removed. How can I do this and how could I print how many rows are dropped? I would be very happy about help.  How can you recognize duplicates? If the columns (action, name) are identical and the time difference is less than or equal to 5 minutes.   ! The time format is 01.10.2019, 9:56:52 date and time are splitted by a comma import pandas as pd d = {'Time': ['01.10.2019, 9:56:52', '01.10.2019, 9:57:15', '02.10.2019 12:56:12', '02.10.2019 13:02:58', '02.10.2019 13:11:58'] ,'Action': ['Opened', 'Opened', 'Closed', 'Opened', 'Opened'] ,'Name': ['Max', 'Max', 'Susan', 'Michael', 'Michael']} df = pd.DataFrame(data=d) display(df.head())  Output  Desired Output  Details 

Answer: IIUC you can create a group number by getting the time difference, and then groupby and first: print (df.assign(group=pd.to_datetime(df["Time"]).diff().dt.seconds.gt(300).cumsum()) .groupby(["group", "Action", "Name"]).first()) Time Action Name group 0 01.10.2019, 9:56:52 Opened Max 1 02.10.2019 12:56:12 Closed Susan 2 02.10.2019 13:02:58 Opened Michael 3 02.10.2019 13:11:58 Opened Michael
Question: I want to open multiple urls at once using Playwright for Python. But I am struggling to figure out how. This is from the async documentation: async def main(): async with async_playwright() as p: for browser_type in [p.chromium, p.firefox, p.webkit]: browser = await browser_type.launch() page = await browser.newPage() await page.goto("https://scrapingant.com/") await page.screenshot(path=f"scrapingant-{browser_type.name}.png") await browser.close() asyncio.get_event_loop().run_until_complete(main())  This opens each browser_type sequentially. How would I go about it if I wanted to do it in parallel? And how would I go about it if I wanted to do something similar with a list of urls? I tried doing this: urls = [ "https://scrapethissite.com/pages/ajax-javascript/#2015", "https://scrapethissite.com/pages/ajax-javascript/#2014", ] async def main(url): async with async_playwright() as p: browser = await p.chromium.launch(headless=False) page = await browser.newPage() await page.goto(url) await browser.close() async def go_to_url(): tasks = [main(url) for url in urls] await asyncio.wait(tasks) go_to_url()  But that gave me the following error: 92: RuntimeWarning: coroutine 'go_to_url' was never awaited go_to_url() RuntimeWarning: Enable tracemalloc to get the object allocation traceback 

Answer: I believe you need to call your go_to_url function using the same recipe: asyncio.get_event_loop().run_until_complete(go_to_url())
Question: I am getting issue with websocket connection as it is getting closed due to handshake error. The error message is as below: WebSocket HANDSHAKING /ws/polData/ [127.0.0.1:59304] Exception inside application: object.__init__() takes exactly one argument (the instance to initialize) Traceback (most recent call last): File "/usr/lib/python3.7/site-packages/channels/routing.py", line 71, in __call__ return await application(scope, receive, send) File "/usr/lib/python3.7/site-packages/channels/sessions.py", line 47, in __call__ return await self.inner(dict(scope, cookies=cookies), receive, send) File "/usr/lib/python3.7/site-packages/channels/sessions.py", line 172, in __call__ return await self.inner(self.scope, receive, self.send) File "/usr/lib/python3.7/site-packages/channels/auth.py", line 181, in __call__ return await super().__call__(scope, receive, send) File "/usr/lib/python3.7/site-packages/channels/middleware.py", line 26, in __call__ return await self.inner(scope, receive, send) File "/usr/lib/python3.7/site-packages/channels/routing.py", line 160, in __call__ send, File "/usr/local/lib/python3.7/dist-packages/asgiref/compatibility.py", line 33, in new_application instance = application(scope) File "/usr/lib/python3.7/site-packages/channels/generic/websocket.py", line 159, in __init__ super().__init__(*args, **kwargs) TypeError: object.__init__() takes exactly one argument (the instance to initialize) WebSocket DISCONNECT /ws/polData/ [127.0.0.1:59304]\ 

Answer: Check your routing.py in app, under websocket_urlpatterns, on repath you may have missed .as_asgi() eg:- websocket_urlpatterns = [ re_path(r'ws/chat/(?P<room_name>\w+)/$', consumers.ChatConsumer.as_asgi()), ]
Question: __repr__ is used to return a string representation of an object, but in Python a function is also an object itself, and can have attributes. How do I set the __repr__ of a function? I see here that an attribute can be set for a function outside the function, but typically one sets a __repr__ within the object definition itself, so I'd like to set the repr within the function definition itself.  My use case is that I am using tenacity to retry a networking function with exponential backoff, and I want to log the (informative) name of the function I have called last. retry_mysql_exception_types = (InterfaceError, OperationalError, TimeoutError, ConnectionResetError) def return_last_retry_outcome(retry_state): """return the result of the last call attempt""" return retry_state.outcome.result() def my_before_sleep(retry_state): print("Retrying {}: attempt {} ended with: {}\n".format(retry_state.fn, retry_state.attempt_number, retry_state.outcome)) @tenacity.retry(wait=tenacity.wait_random_exponential(multiplier=1, max=1200), stop=tenacity.stop_after_attempt(30), retry=tenacity.retry_if_exception_type(retry_mysql_exception_types), retry_error_callback=return_last_retry_outcome, before_sleep=my_before_sleep) def connect_with_retries(my_database_config): connection = mysql.connector.connect(**my_database_config) return connection  Currently retry_state.fn displays something like <function <lambda> at 0x1100f6ee0> like @chepner says, but I'd like to add more information to it.

Answer: I think a custom decorator could help: import functools class reprable: """Decorates a function with a repr method. Example: >>> @reprable ... def foo(): ... '''Does something cool.''' ... return 4 ... >>> foo() 4 >>> foo.__name__ 'foo' >>> foo.__doc__ 'Does something cool.' >>> repr(foo) 'foo: Does something cool.' >>> type(foo) <class '__main__.reprable'> """ def __init__(self, wrapped): self._wrapped = wrapped functools.update_wrapper(self, wrapped) def __call__(self, *args, **kwargs): return self._wrapped(*args, **kwargs) def __repr__(self): return f'{self._wrapped.__name__}: {self._wrapped.__doc__}'  Demo: http://tpcg.io/uTbSDepz.
Question: When I install opencv-python-headless in Google Colab, it takes 15 minutes to complete. My code: ! pip install --upgrade pip ! pip install opencv-python-headless  Here's a notebook with this code which recreates the problem: https://colab.research.google.com/gist/mherzog01/38b6cf71942a443da072f09bc097387f/slow-install-of-opencv-python-headless.ipynb. The process eventually completes, but I'd like to reduce the time to install. I saw from `Building wheel for opencv-python (PEP 517) ... -` runs forever a discussion about compiling OpenCV, which could very well be what is happening here. However, this same SO post states that if you upgrade pip, it will use pre-built wheels. Edit: Added @intsco's workaround to the Google Colab

Answer: Might be related to changes in OpenCV >=4.3 wheels https://github.com/skvark/opencv-python#backward-compatibility  Starting from 4.3.0 and 3.4.10 builds the Linux build environment was updated from manylinux1 to manylinux2014. This dropped support for old Linux distributions.  My workaround: pip install "opencv-python-headless<4.3"
Question: Once I have downloaded a pre-trained model on a Colab Notebook, it disappears after I reset the notebook variables. Is there a way I can download the model to use it for a second occasion? tokenizer = BertTokenizer.from_pretrained('bert-base-uncased') 

Answer: Mount your google drive: from google.colab import drive drive.mount('/content/drive')  Do your stuff and save your models: from transformers import BertTokenizer tokenizer = BertTokenizer.from_pretrained('bert-base-uncased') tokenizer.save_pretrained('/content/drive/My Drive/tokenizer/')  Reload it in a new session: tokenizer2 = BertTokenizer.from_pretrained('/content/drive/My Drive/tokenizer/')
Question: I try to get the id of certain HTML tags using Python and Selenium. There is html code: <tr id="10"> <td colspan="5"> <div class="card-view"> <span class="value">PROVIDER_628_54678931</span> </div> </td> </tr> <tr id="11"> <td colspan="5"> <div class="card-view"> <span class="value">PROVIDER_629_54678932</span> </div> </td> </tr> <tr id="12"> <td colspan="5"> <div class="card-view"> <span class="value">PROVIDER_730_54678933</span> </div> </td> </tr> <tr id="13"> <td colspan="5"> <div class="card-view"> <span class="value">PROVIDER_6542_54678934</span> </div> </td> </tr>  For extract id of only one parent tag i do : elem = browser.find_element_by_xpath("//span[contains(@class, 'value') and text()='PROVIDER_628_54678931']") parent = elem.find_element_by_xpath('../../..') print(parent.get_attribute("id"))  How to use regex in XPath to get parent id-s of "span" element where the text contains "PROVIDER_6XX", but not "PROVIDER_7" and PROVIDER_6542?

Answer: I found solution here : link def findTrunksByRegExp(): pattern = re.compile(r"PROVIDER_6\d{2}") elements = browser.find_elements_by_xpath("//span[contains(@class, 'value')]") for element in elements: match = pattern.match(element.text) if match: parent = element.find_element_by_xpath('../../..') print(parent.get_attribute("id"))
Question: The matplotlib color palettes often feature white or very light colors which do not show up well on scatter or line plots. I am making a plot in which I use norm = mpl.colors.Normalize(vmin=0, vmax=1) cmap = mpl.cm.ScalarMappable(norm=norm, cmap=mpl.cm.Blues) plt.plot(x, y, c=cmap.to_rgba(z)) cbar = plt.colorbar(cmap)  to plot numerous lines. I want to modify the color palette to remove the first, say, 30% of the colors. How can this be achieved?  My solution, which is fairly poor, is to modify two of the lines as follows: norm = mpl.colors.Normalize(vmin=-0.4, vmax=1) cbar = plt.colorbar(cmap, boundaries=[0,0.2,0.4,0.6,0.8,1]) 

Answer: You can easily create a custom colormap using LinearSegmentedColormap and choosing the colors that you want (in this case, a subset of the original colormap) min_val, max_val = 0.3,1.0 n = 10 orig_cmap = plt.cm.Blues colors = orig_cmap(np.linspace(min_val, max_val, n)) cmap = matplotlib.colors.LinearSegmentedColormap.from_list("mycmap", colors)  demo gradient = np.linspace(0, 1, 256) gradient = np.vstack((gradient, gradient)) fig, (ax1, ax2) = plt.subplots(2,1, figsize=(5,2)) ax1.imshow(gradient, cmap=orig_cmap, aspect='auto') ax1.set_title('original') ax2.imshow(gradient, cmap=cmap, aspect='auto') ax2.set_title('custom') plt.tight_layout()
Question: I would like to unpivot a DataFrame with MultiIndex columns, but I struggle to get the exact output I want. I played with all the parameters of the pd.melt() function but couldn't make it... Here is the kind of input I have : import pandas as pd indexes = [['TC1', 'TC2'], ['x', 'z', 'Temp']] data = pd.DataFrame(columns=pd.MultiIndex.from_product(indexes)) data.loc[0,('TC1', 'x')] = 10 data.loc[0,('TC1', 'z')] = 100 data.loc[0,('TC1', 'Temp')] = 250 data.loc[0,('TC2', 'x')] = 20 data.loc[0,('TC2', 'z')] = 200 data.loc[0,('TC2', 'Temp')] = 255  And here is the kind of output I would like, with "Time" column being the index of data  Time TC x z Temp 0 0 TC1 10 100 250 1 0 TC2 20 200 255  My real data have far more columns of kind TCx. Any clue?

Answer: Give this a try data_out = data.stack(level=0).rename_axis(['Time','TC']).reset_index() Out[87]: Time TC Temp x z 0 0 TC1 250 10 100 1 0 TC2 255 20 200
Question: I want to convert images to tensor using torchvision.transforms.ToTensor(). After processing, I printed the image but the image was not right. Here is my code: trans = transforms.Compose([ transforms.ToTensor()]) demo = Image.open(img) demo_img = trans(demo) demo_array = demo_img.numpy()*255 print(Image.fromarray(demo_array.astype(np.uint8)))  The original image is:  After processing, it looks like: 

Answer: It seems that the problem is with the channel axis. If you look at torchvision.transforms docs, especially on ToTensor()  Converts a PIL Image or numpy.ndarray (H x W x C) in the range [0, 255] to a torch.FloatTensor of shape (C x H x W) in the range [0.0, 1.0]  So once you perform the transformation and return to numpy.array your shape is: (C, H, W) and you should change the positions, you can do the following: demo_array = np.moveaxis(demo_img.numpy()*255, 0, -1)  This will transform the array to shape (H, W, C) and then when you return to PIL and show it will be the same image. So in total: import numpy as np from PIL import Image from torchvision import transforms trans = transforms.Compose([transforms.ToTensor()]) demo = Image.open(img) demo_img = trans(demo) demo_array = np.moveaxis(demo_img.numpy()*255, 0, -1) print(Image.fromarray(demo_array.astype(np.uint8)))
Question: After having tried many things, I thought it would be good to ask on SO. My problem is fairly simple: how can I solve the following equation using Sympy? Equation I want to solve this for lambda_0 and q is an array of size J containing elments between 0 and 1 that sum op to 1 (discrete probability distribution). I tried the following: from sympy.solvers import solve from sympy import symbols, summation p = [0.2, 0.3, 0.3, 0.1, 0.1] l = symbols('l') j = symbols('j') eq= summation(j*q[j]/(l-j), (j, 0, 4)) s= solve(eq, l)  But this gives me an error for q[j] as j is a Symbol object here and not an integer. If I don't make j as symbol, I cannot evaluate the eq expression. Does anyone know how to do this? Edit: p = 1-q in the above, hence q[j] should have been replaced by (1-p[j]).

Answer: List p needs to be converted into symbolic array before it can be indexed with symbolic value j. from sympy.solvers import solve from sympy import symbols, summation, Array p = Array([0.2, 0.3, 0.3, 0.1, 0.1]) l, j = symbols('l j') eq = summation(j * (1 - p[j]) / (l - j), (j, 0, 4)) s = solve(eq - 1, l) # [1.13175762143963 + 9.29204634892077e-30*I, 2.23358705810004 - 1.36185313905566e-29*I, 3.4387382449005 + 3.71056356734273e-30*I, 11.5959170755598 + 6.15921474293073e-31*I]  (assuming your p stands for 1 - q)
Question: I need to add a number of columns (4000) into the data frame in pyspark. I am using the withColumn function, but getting assertion error. df3 = df2.withColumn("['ftr' + str(i) for i in range(0, 4000)]", [expr('ftr[' + str(x) + ']') for x in range(0, 4000)])   Not sure what is wrong.

Answer: Try to do something like this: df2 = df3 for i in range(0, 4000): df2 = df2.withColumn(f"ftr{i}", lit(f"frt{i}"))
Question: Is there any difference between torch.optim.Adam(weight_decay=0.01) and torch.optim.AdamW(weight_decay=0.01)? Link to the docs: torch.optim.

Answer: Yes, Adam and AdamW weight decay are different.  Hutter pointed out in their paper (Decoupled Weight Decay Regularization) that the way weight decay is implemented in Adam in every library seems to be wrong, and proposed a simple way (which they call AdamW) to fix it.  In Adam, the weight decay is usually implemented by adding wd*w (wd is weight decay here) to the gradients (Ist case), rather than actually subtracting from weights (IInd case). # Ist: Adam weight decay implementation (L2 regularization) final_loss = loss + wd * all_weights.pow(2).sum() / 2 # IInd: equivalent to this in SGD w = w - lr * w.grad - lr * wd * w   These methods are same for vanilla SGD, but as soon as we add momentum, or use a more sophisticated optimizer like Adam, L2 regularization (first equation) and weight decay (second equation) become different.  AdamW follows the second equation for weight decay. In Adam  weight_decay (float, optional) – weight decay (L2 penalty) (default: 0)  In AdamW  weight_decay (float, optional) – weight decay coefficient (default: 1e-2)  Read more on the fastai blog.
Question: I have a problem when I want to paginate the filter that I create with django_filter, in my template it shows me the query set and filter but paginate does not work, I would like to know why this happens and if you could help me. I'll insert snippets of my code so you can see. This is my views.py PD: i have all the necesary imports. @method_decorator(staff_member_required, name='dispatch') class EmployeeListView(ListView): model = Employee paginate_by = 4 def dispatch(self, request, *args, **kwargs): if not request.user.has_perm('employee.view_employee'): return redirect(reverse_lazy('home')) return super(EmployeeListView, self).dispatch(request, *args, **kwargs) def get_context_data(self, **kwargs): context = super().get_context_data(**kwargs) context['filter'] = EmployeeFilter(self.request.GET, queryset = self.get_queryset()) return context  filters.py import django_filters from .models import Employee, Accident class EmployeeFilter(django_filters.FilterSet): class Meta: model = Employee fields = { 'rutEmployee' : ['startswith'] } 

Answer: You should override get_queryset.This means you have to put your filter in get_queryset like this: @method_decorator(staff_member_required, name='dispatch') class EmployeeListView(ListView): model = Employee paginate_by = 4 def dispatch(self, request, *args, **kwargs): if not request.user.has_perm('employee.view_employee'): return redirect(reverse_lazy('home')) return super(EmployeeListView, self).dispatch(request, *args, **kwargs) def get_context_data(self, **kwargs): context = super().get_context_data(**kwargs) context['filter'] = EmployeeFilter(self.request.GET, queryset = self.get_queryset()) return context def get_queryset(self): queryset = super().get_queryset() return EmployeeFilter(self.request.GET, queryset=queryset).qs  and use object_list instead of filter in employee_list.html like this: {% for employee in object_list|dictsort:"id" reversed %}
Question: I can run my tests by executing (on Windows) pytest .\tests\test_x.py  Result: ================================= test session starts ================================== platform win32 -- Python 3.8.3, pytest-5.4.3, py-1.9.0, pluggy-0.13.1 rootdir: C:\Users\...... collected 9 items tests\test_x.py ......... [100%] ================================== 9 passed in 3.67s ===================================  However, the following two commands pytest -m tests pytest -m test  got the following result. Why all the tests are deselected while they can be run as script? PS C:\Users\......> pytest -m test ================================= test session starts ================================== platform win32 -- Python 3.8.3, pytest-5.4.3, py-1.9.0, pluggy-0.13.1 rootdir: C:\Users\...... collected 9 items / 9 deselected ================================ 9 deselected in 3.78s ================================= 

Answer: You're using -m which filters which tests to run according to how you mark your tests. You're telling pytest to only run tests tagged @pytest.mark.test. Presumably, you don't have any tests marked as such. https://docs.pytest.org/en/stable/example/markers.html#mark-run