eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: For simplicity to parse/create JSON, machine learning applications usually uses the Bunch object, e.g. https://github.com/dsc/bunch/blob/master/bunch/__init__.py When getting, there's a nested EAFP idiom that checks through the dict.get() function and then trying to access it with dictionary square bracket syntax, i.e. class Bunch(dict): def __getattr___(self, k): try: return object.__getattribute__(self, k) except AttributeError: try: return self[k] except KeyError: raise AttributeError  And when trying to set an attribute,  def __setattr__(self, k, v): try: # Throws exception if not in prototype chain object.__getattribute__(self, k) except AttributeError: try: self[k] = v except: raise AttributeError(k) else: object.__setattr__(self, k, v)  Seems like the sklearn implementation follows the same train of thought but has lesser checks https://github.com/scikit-learn/scikit-learn/blob/2beed5584/sklearn/utils/__init__.py#L61 class Bunch(dict): def __init__(self, **kwargs): super().__init__(kwargs) def __setattr__(self, key, value): self[key] = value def __dir__(self): return self.keys() def __getattr__(self, key): try: return self[key] except KeyError: raise AttributeError(key) def __setstate__(self, state): # Bunch pickles generated with scikit-learn 0.16.* have an non # empty __dict__. This causes a surprising behaviour when # loading these pickles scikit-learn 0.17: reading bunch.key # uses __dict__ but assigning to bunch.key use __setattr__ and # only changes bunch['key']. More details can be found at: # https://github.com/scikit-learn/scikit-learn/issues/6196. # Overriding __setstate__ to be a noop has the effect of # ignoring the pickled __dict__ pass  The nested EAFP seems a little hard to maintain, my questions here are:  Is there a simpler way to handle get and set functions for Bunch data objects? Are there any other Dict like object that allows mutability between attributes and keys? How should Bunch object's .update() function work, shallow or deep copying? Or just let the default dict.update() do what it does? Understanding dict.copy() - shallow or deep? 

Answer: Lucky for you, all objects have an internal dict-like object that manages the attributes of the object (this is in the __dict__ attribute). To do what you're asking, you just need to make the class use itself as the __dict__ object: class Bunch(dict): def __init__(self, *args, **kwargs): self.__dict__ = self super().__init__(*args, **kwargs)  Usage: >>> b = Bunch() >>> b.foo = 3 >>> b["foo"] 3 >>> b["foo"] = 5 >>> b.foo 5 >>> b["bar"] = 1 >>> b.bar 1
Question: I've been getting this error on several programs for now. I've tried upgrading pytube, reinstalling it, tried some fixes, changed URLs and code, but nothing seems to work. from pytube import YouTube #ask for the link from user link = input("Enter the link of YouTube video you want to download: ") yt = YouTube(link) #Showing details print("Title: ",yt.title) print("Number of views: ",yt.views) print("Length of video: ",yt.length) print("Rating of video: ",yt.rating) #Getting the highest resolution possible ys = yt.streams.get_highest_resolution() #Starting download print("Downloading...") ys.download() print("Download completed!!")  and this is the error code:  File "C:\Users\Madjid\PycharmProjects\pythonProject\app2.py", line 6, in <module> yt = YouTube(link) File "C:\Users\Madjid\PycharmProjects\pythonProject\venv\lib\site-packages\pytube\__main__.py", line 91, in __init__ self.prefetch() File "C:\Users\Madjid\PycharmProjects\pythonProject\venv\lib\site-packages\pytube\__main__.py", line 181, in prefetch self.vid_info_raw = request.get(self.vid_info_url) File "C:\Users\Madjid\PycharmProjects\pythonProject\venv\lib\site-packages\pytube\request.py", line 36, in get return _execute_request(url).read().decode("utf-8") File "C:\Users\Madjid\PycharmProjects\pythonProject\venv\lib\site-packages\pytube\request.py", line 24, in _execute_request return urlopen(request) # nosec File "E:\Python\lib\urllib\request.py", line 214, in urlopen return opener.open(url, data, timeout) File "E:\Python\lib\urllib\request.py", line 523, in open response = meth(req, response) File "E:\Python\lib\urllib\request.py", line 632, in http_response response = self.parent.error( File "E:\Python\lib\urllib\request.py", line 555, in error result = self._call_chain(*args) File "E:\Python\lib\urllib\request.py", line 494, in _call_chain result = func(*args) File "E:\Python\lib\urllib\request.py", line 747, in http_error_302 return self.parent.open(new, timeout=req.timeout) File "E:\Python\lib\urllib\request.py", line 523, in open response = meth(req, response) File "E:\Python\lib\urllib\request.py", line 632, in http_response response = self.parent.error( File "E:\Python\lib\urllib\request.py", line 561, in error return self._call_chain(*args) File "E:\Python\lib\urllib\request.py", line 494, in _call_chain result = func(*args) File "E:\Python\lib\urllib\request.py", line 641, in http_error_default raise HTTPError(req.full_url, code, msg, hdrs, fp) urllib.error.HTTPError: HTTP Error 410: Gone 

Answer: Try to upgrade, there is a fix in version 11.0.0: python -m pip install --upgrade pytube
Question: I have deployed a Django project using Apache2, everything is working fine except for weazyprint which creates PDF file for forms. The pdf was working fine in testing and local host. Now everytime I access the pdf it is showing this error: FileNotFoundError at /business_plan/businessplan/admin/info/2/pdf/ [Errno 2] No such file or directory: '/home/ahesham/Portfolio/static\\css\\report.css'  I have tried to change the \ and adding it twice but it didn't work here is the views.py def admin_order_pdf(request, info_id): info = get_object_or_404(Info, id=info_id) html = render_to_string('businessplan/pdf.html', {'info': info}) response = HttpResponse(content_type='application/pdf') response['Content-Disposition'] = 'filename="order_{}.pdf"'.format( Info.id) weasyprint.HTML(string=html,base_url=request.build_absolute_uri()).write_pdf(response, stylesheets=[weasyprint.CSS(settings.STATICFILES_DIRS[0] + '\css\\report.css')], presentational_hints=True) return response  The error is comming from this \css\\report.css knowing that the report.css file is in the static folder and all css and js of the deplyed site is working perfectly fine and I tried python manage.py collectstatic did not work. I am not sure exactly why is the error showing if it is because of ubunto or django views. Update: I have tried changing the location to be as following: stylesheets=[weasyprint.CSS(settings.STATICFILES_DIRS[0] + '/css/report.css')], presentational_hints=True)  This is the error that appeared function/symbol 'pango_context_set_round_glyph_positions' not found in library 'libpango-1.0.so.0': /usr/lib/x86_64-linux-gnu/libpango-1.0.so.0: undefined symbol: pango_context_set_round_glyph_positions  So I have searched for the solution and I tried to download this package: sudo apt-get install libpango1.0-0  and also tried: install libpango1.0-dev  but still nothing changed it wouldn't work getting the same error. I have also tried replacing static directory with static root as the project is deployed buy i got the same error function/symbol 'pango_context_set_round_glyph_positions' the following: stylesheets=[weasyprint.CSS(settings.STATIC_ROOT + '/css/report.css')]  This is the settings file: STATIC_URL = '/static/' #if DEBUG: # STATICFILES_DIRS = [os.path.join(BASE_DIR, 'static')] #else: #STATIC_ROOT = os.path.join(BASE_DIR, 'static') STATICFILES_DIRS = [os.path.join(BASE_DIR, 'static' )] MEDIA_URL = '/media/' MEDIA_ROOT = os.path.join(BASE_DIR, 'media')  I am using Ubuntu 18.04 LTS Disk - Apache2 Server My question: What am I doing wrong and how do I fix it? Please let me know if additional information required to help fix this error. Any feedback will be highly appreciated if there are other ways to display PDF in Django admin

Answer: I had also faced the same error  'pango_context_set_round_glyph_positions' not found in library  i think you must be using weasyprint version 53.0 which requires pango version 1.44.0+ downgrading the weasyprint version to 52.5 solved my issue.because this does not require recent version of pango. you can also check this https://github.com/Kozea/WeasyPrint/issues/1384
Question: I have the following dataset: df_plots = pd.DataFrame({'Group':['A','A','A','A','A','A','B','B','B','B','B','B'], 'Type':['X','X','X','Y','Y','Y','X','X','X','Y','Y','Y'], 'Value':[1,1.2,1.4,1.3,1.8,1.5,15,19,18,17,12,13]}) df_plots Group Type Value 0 A X 1.0 1 A X 1.2 2 A X 1.4 3 A Y 1.3 4 A Y 1.8 5 A Y 1.5 6 B X 15.0 7 B X 19.0 8 B X 18.0 9 B Y 17.0 10 B Y 12.0 11 B Y 13.0  And I want to create boxplots per Group (there are two in the example) and in each plot to show by type. I have tried this: fig, axs = plt.subplots(1,2,figsize=(8,6), sharey=False) axs = axs.flatten() for i, g in enumerate(df_plots[['Group','Type','Value']].groupby(['Group','Type'])): g[1].boxplot(ax=axs[i])   Results in an IndexError, because the loop tries to create 4 plots.  --------------------------------------------------------------------------- IndexError Traceback (most recent call last) <ipython-input-12-8e1150950024> in <module> 3 4 for i, g in enumerate(df[['Group','Type','Value']].groupby(['Group','Type'])): ----> 5 g[1].boxplot(ax=axs[i]) IndexError: index 2 is out of bounds for axis 0 with size 2  Then I tried this: fig, axs = plt.subplots(1,2,figsize=(8,6), sharey=False) axs = axs.flatten() for i, g in enumerate(df_plots[['Group','Type','Value']].groupby(['Group','Type'])): g[1].boxplot(ax=axs[i], by=['Group','Type'])  But no, I have the same problem. The expected result should have only two plots, and each plot have a box-and-whisker per Type. This is a sketch of this idea:  Please, any help will be greatly appreciated, with this code I can control some aspects of the data that I can't with seaborn.

Answer: As @Prune mentioned, the immediate issue is that your groupby() returns four groups (AX, AY, BX, BY), so first fix the indexing and then clean up a couple more issues:  Change axs[i] to axs[i//2] to put groups 0 and 1 on axs[0] and groups 2 and 3 on axs[1]. Add positions=[i] to place the boxplots side by side rather than stacked. Set the title and xticklabels after plotting (I'm not aware of how to do this in the main loop).  for i, g in enumerate(df_plots.groupby(['Group', 'Type'])): g[1].boxplot(ax=axs[i//2], positions=[i]) for i, ax in enumerate(axs): ax.set_title('Group: ' + df_plots['Group'].unique()[i]) ax.set_xticklabels(['Type: X', 'Type: Y'])    Note that mileage may vary depending on version:      matplotlib.__version__ pd.__version__     confirmed working 3.4.2 1.3.1   confirmed not working 3.0.1 1.2.4
Question: I have installed many many packages including torch, gpytorch, ... in the past in Windows, Ubuntu and Mac following this scenario: conda create -n env_name conda activate env_name conda install pytorch torchvision torchaudio cudatoolkit=11.1 -c pytorch -c nvidia  However, this time on Ubuntu, I interfered the following error on downloading the package which apparently after downloading when checking the checksum, it sees a mismatch. I also tried removing those *.bz2 files just in case if there is a pre-downloaded file, it didn't work. ChecksumMismatchError: Conda detected a mismatch between the expected content and downloaded content for url 'https://conda.anaconda.org/pytorch/linux-64/torchaudio-0.9.0-py39.tar.bz2'. download saved to: /home/amin/anaconda3/pkgs/torchaudio-0.9.0-py39.tar.bz2 expected md5: 7224453f68125005e034cb6646f2f0a3 actual md5: 6bbb8056603453427bbe4cca4b033361 ChecksumMismatchError: Conda detected a mismatch between the expected content and downloaded content for url 'https://conda.anaconda.org/pytorch/linux-64/torchvision-0.10.0-py39_cu111.tar.bz2'. download saved to: /home/amin/anaconda3/pkgs/torchvision-0.10.0-py39_cu111.tar.bz2 expected md5: 78b4c927e54b06d7a6d18eec8b3f2d18 actual md5: 69dd8411c573903db293535017742bd9  My system information: Linux SPOT-Server 5.8.0-63-generic #71~20.04.1-Ubuntu SMP Thu Jul 15 17:46:08 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux  My conda --version is also 4.8.2. I also should add that I have same issue on Windows having conda --version equal to 4.10.1.

Answer: The PyTorch channel maintainers had an issue when uploading some new package builds, which has since been resolved (see GitHub Issue). The technical cause was uploading new builds with identical versions and build numbers as before, without replacing the previous build. This caused the expected MD5 checksum to correspond to the new upload, but the tarball that was ultimately downloaded still corresponded to the previous upload, leading to a checksum mismatch.
Question: I'd like to select one column only but all the rows except last row. If I did it like below, the result is empty. a = data_vaf.loc[:-1, 'Area'] 

Answer: loc:location iloc:index location. They just can't operate implicitly. Therefore we exclude last raw by iloc then select the column Area As shown by the comment from @ThePyGuy data_vaf.iloc[:-1]['Area']   Here's the structure of iloc[row, column] And iloc[row] do the same thing as iloc[row,:] df.iloc[:-1] do the same thing as df[:-1]
Question: Traceback Error Traceback (most recent call last): File "C:\Users\trial2\trial.py", line 55, in <module> image_stack(image) File "C:\Users\trial2\trial.py", line 41, in image_stack transposed_axes = np.transpose(img, axes=concat) File "<__array_function__ internals>", line 5, in transpose File "C:\Users\trial2\venv\lib\site-packages\numpy\core\fromnumeric.py", line 660, in transpose return _wrapfunc(a, 'transpose', axes) File "C:\Users\trial2\venv\lib\site-packages\numpy\core\fromnumeric.py", line 57, in _wrapfunc return bound(*args, **kwds) ValueError: axes don't match array  The Traceback Error is also there, if that helps to solve the issue I need to calculate the new transpose axes for the image that has been inputted into the function as parameter. The function gets the shape of the image, and then calculates the transpose axes for the image. The problem is occurring in transposing the image array to the new axes. I can't figure out the issue to why it isn't working. Do I need to turn the concat variable into a tuple for it to work properly or is something else causing the issue for it def image_stack(image): img_shape = cv2.imread(image).shape img = np.asarray(img_shape) print(type(img)) n = len(img) val_1 = list(range(1, n - 1, 2)) val_2 = list(range(0, n - 1, 2)) print(img) if n % 2 == 0: y_ax = val_1 x_ax = val_2 axes = (y_ax, x_ax, [n-1]) else: y_ax = val_2 x_ax = val_1 axes = (y_ax, x_ax, [n - 1]) print(type(axes)) concat = np.concatenate(axes) print(concat) if type(axes) == tuple: transposed_axes = np.transpose(img, axes=concat) print(transposed_axes) image = 'C:\\trial_images\\9.jpg' image_stack(image) 

Answer: The type of the axes argument doesn't matter: In [96]: arr = np.ones( [720, 1280, 3 ] ) In [97]: np.transpose(arr,[0,1,2]).shape Out[97]: (720, 1280, 3) In [98]: np.transpose(arr,(0,1,2)).shape Out[98]: (720, 1280, 3) In [99]: np.transpose(arr,np.array([0,1,2])).shape Out[99]: (720, 1280, 3)  but if I provide more values than dimensions of arr I get your error: In [103]: np.transpose(arr,[0,1,2,3]).shape Traceback (most recent call last): File "<ipython-input-103-fc01ec77b59e>", line 1, in <module> np.transpose(arr,[0,1,2,3]).shape File "<__array_function__ internals>", line 5, in transpose File "/usr/local/lib/python3.8/dist-packages/numpy/core/fromnumeric.py", line 660, in transpose return _wrapfunc(a, 'transpose', axes) File "/usr/local/lib/python3.8/dist-packages/numpy/core/fromnumeric.py", line 57, in _wrapfunc return bound(*args, **kwds) ValueError: axes don't match array  I expect the same for [0,1,2] and a 2d arr. Hopefully this gives you ideas of how to debug problems yourself. Fireup an interactive session and try some alternatives.
Question: Following is a small working code for Stochastic Weight Averaging in Pytorch taken from here. loader, optimizer, model, loss_fn = ... swa_model = torch.optim.swa_utils.AveragedModel(model) scheduler = torch.optim.lr_scheduler.CosineAnnealingLR(optimizer, T_max=300) swa_start = 160 swa_scheduler = SWALR(optimizer, swa_lr=0.05) for epoch in range(300): for input, target in loader: optimizer.zero_grad() loss_fn(model(input), target).backward() optimizer.step() if epoch > swa_start: swa_model.update_parameters(model) swa_scheduler.step() else: scheduler.step() # Update bn statistics for the swa_model at the end torch.optim.swa_utils.update_bn(loader, swa_model) # Use swa_model to make predictions on test data preds = swa_model(test_input)  In this code after 160th epoch the swa_scheduler is used instead of the usual scheduler. What does swa_lr signify? The documentation says,  Typically, in SWA the learning rate is set to a high constant value. SWALR is a learning rate scheduler that anneals the learning rate to a fixed value, and then keeps it constant.   So what happens to the learning rate of the optimizer after 160th epoch? Does swa_lr affect the optimizer learning rate?  Suppose that at the beginning of the code the optimizer was ADAM initialized with a learning rate of 1e-4. Then does the above code imply that for the first 160 epochs the learning rate for training will be 1e-4 and then for the remaining number of epochs it will be swa_lr=0.05? If yes, is it a good idea to define swa_lr also to 1e-4?

Answer:    does the above code imply that for the first 160 epochs the learning rate for training will be 1e-4  No it won't be equal to 1e-4, during the first 160 epochs the learning rate is managed by the first scheduler scheduler. This one is a initialize as a torch.optim.lr_scheduler.CosineAnnealingLR. The learning rate will follow this curve:        for the remaining number of epochs it will be swa_lr=0.05  This is partially true, during the second part - from epoch 160 - the optimizer's learning rate will be handled by the second scheduler swa_scheduler. This one is initialized as a torch.optim.swa_utils.SWALR. You can read on the documentation page:  SWALR is a learning rate scheduler that anneals the learning rate to a fixed value [swa_lr], and then keeps it constant.  By default (cf. source code), the number of epochs before annealing is equal to 10. Therefore the learning rate from epoch 170 to epoch 300 will be equal to swa_lr and will stay this way. The second part will be:  This complete profile, i.e. both parts:        If yes, is it a good idea to define swa_lr also to 1e-4  It is mentioned in the docs:  Typically, in SWA the learning rate is set to a high constant value.  Setting swa_lr to 1e-4 would result in the following learning-rate profile:
Question: the following code has 5 workers .... each opens its own worker_task() with concurrent.futures.ThreadPoolExecutor(max_workers=5) as executor: future_to_url = {executor.submit(worker_task, command_, site_): site_ for site_ in URLS} for future in concurrent.futures.as_completed(future_to_url): url = future_to_url[future] try: data = future.result()  BUT ..... inside each worker_task() ...... I cannot identify ... which of the 5 workers is currently being used (Worker_ID) If I want to print('worker 3 has finished') inside worker_task() ..... I cannot do this because executor.submit does not allow Any solutions?

Answer: You can get name of worker thread with the help of threading.current_thread() function. Please find some example below: from concurrent.futures import ThreadPoolExecutor, Future from threading import current_thread from time import sleep from random import randint # imagine these are urls URLS = [i for i in range(100)] def do_some_work(url, a, b): """Simulates some work""" sleep(2) rand_num = randint(a, b) if rand_num == 5: raise ValueError("No! 5 found!") r = f"{current_thread().getName()}||: {url}_{rand_num}\n" return r def show_fut_results(fut: Future): """Callback for future shows results or shows error""" if not fut.exception(): print(fut.result()) else: print(f"{current_thread().getName()}| Error: {fut.exception()}\n") if __name__ == '__main__': with ThreadPoolExecutor(max_workers=10) as pool: for i in URLS: _fut = pool.submit(do_some_work, i, 1, 10) _fut.add_done_callback(show_fut_results)  If you want more control over threads, use threading module: from threading import Thread from queue import Queue from time import sleep from random import randint import logging # imagine these are urls URLS = [f"URL-{i}" for i in range(100)] # number of worker threads WORKER_NUM = 10 def do_some_work(url: str, a: int, b: int) -> str: """Simulates some work""" sleep(2) rand_num = randint(a, b) if rand_num == 5: raise ValueError(f"No! 5 found in URL: {url}") r = f"{url} = {rand_num}" return r def thread_worker_func(q: Queue, a: int, b: int) -> None: """Target function for Worker threads""" logging.info("Started working") while True: try: url = q.get() # if poison pill - stop worker thread if url is None: break r = do_some_work(url, a, b) logging.info(f"Result: {r}") except ValueError as ex: logging.error(ex) except Exception as ex: logging.error(f"Unexpected error: {ex}") logging.info("Finished working") if __name__ == '__main__': logging.basicConfig( level=logging.INFO, format="%(levelname)s | %(threadName)s | %(asctime)s | %(message)s", ) in_q = Queue(50) workers = [ Thread(target=thread_worker_func, args=(in_q, 1, 10, ), name=f"MyWorkerThread-{i+1}") for i in range(WORKER_NUM) ] [w.start() for w in workers] # start distributing tasks for _url in URLS: in_q.put(_url) # send poison pills to worker-threads for w in workers: in_q.put(None) # wait worker thread to join Main Thread logging.info("Main Thread waiting for Worker Threads") [w.join() for w in workers] logging.info("Workers joined") sleep(10) logging.info("App finished")
Question: Why does {'kwargs':{'1':'a', '2':'b'}} appear when I run test_func()? I would have expected just this to print: {'1':'a', '2':'b'}. Code: class MyClass: def __init__(self, **kwargs): self.kwargs = kwargs def test_func(self): print(self.kwargs) test_kwargs = {'1':'a', '2':'b'} my_class = MyClass(kwargs=test_kwargs) my_class.test_func()  Ouput: {'kwargs': {'1': 'a', '2': 'b'}} 

Answer: It's because you initialize the instance by passing 1 keyword argument named kwargs with the dictionary as value. If you want to see the dictionary as kwargs, you need to call in using my_class = MyClass(**test_kwargs)
Question: Why do I need to use a dictionary and set the parameters? Such as lat, lng in following: parameters = { "lat": MY_LAT, "lng": MY_LONG, "formatted": 0 } response = requests.get(url="https://api.sunrise-sunset.org/json", params=parameters)  Why can't I do the below, for instance to put in latitude? response = requests.get(url="https://api.sunrise-sunset.org/json", params=parameters, lat=MY_LAT) 

Answer: You can't pass the arguments by name because the Python requests library doesn't know what parameters a given URL accepts. The API at sunrise-sunset.org takes lat and lng parameters, but most other APIs would have no use for them. By passing a dictionary of key=value pairs, you tell requests both the names and values of the parameters expected by the specific API you're calling.
Question: I like to get inspiration from well designed python projects. The last one that inspired me was the poetry repository. I copied a lot from that, but the subject of this post are black and isort. Both are well configured in pyproject.toml: [tool.isort] profile = "black" ... known_first_party = "poetry" [tool.black] line-length = 88 include = '\.pyi?$' exclude = ''' /( ... )/ '''  and formatting is configured in the Makefileas: format: clean @poetry run black poetry/ tests/  I thought that running make format would run blackwhich would internally run isort, but when I ran isort ., it correctly formated the import statements afterwards. It then seems black did not run isort. Question: does black run isort internally?

Answer:  Question: does black run isort internally?  No, it doesn't. isort has a profile = "black" option that makes it adhere to Black's standards though. The poetry repository itself has a pre-commit hook defined here in .pre-commit-config.yaml that makes sure isort is run (along with a couple of other tools).