eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I started programming using FastAPI framework and it comes with a builtin Swagger interface to handle requests and responses. I have completed nearly 20 APIs and its hard to manage and recognise APIs on Swagger interface. Someone told me to add sections in Swagger interface to distinguish APIs, but I couldn't find any examples and I need help.

Answer: You can add tags to your path parameter, for example. If you have something like this, using tags is extremely helpful. @app.delete("/items", tags=["Delete Methods"]) @app.put("/items", tags=["Put Methods"]) @app.post("/items", tags=["Post Methods"]) @app.get("/items", tags=["Get Methods"]) async def handle_items(): return @app.get("/something", tags=["Get Methods"]) async def something(): return   You will get this, also if you want to add description and you don't want to keep repating yourself (For example adding same description in all parameters) You can use openapi_tags (I prefer this) from fastapi import FastAPI tags_metadata = [ {"name": "Get Methods", "description": "One other way around"}, {"name": "Post Methods", "description": "Keep doing this"}, {"name": "Delete Methods", "description": "KILL 'EM ALL"}, {"name": "Put Methods", "description": "Boring"}, ] app = FastAPI(openapi_tags=tags_metadata) @app.delete("/items", tags=["Delete Methods"]) @app.put("/items", tags=["Put Methods"]) @app.post("/items", tags=["Post Methods"]) @app.get("/items", tags=["Get Methods"]) async def handle_items(): return  This will give the same look without repetition
Question: I am trying to scrape a website. I have tried using two methods but both do not provide me with the full website source code that I am looking for. I am trying to scrape the news titles from the website URL provided below. URL: "https://www.todayonline.com/" These are the two methods I have tried but failed. Method 1: Beautiful Soup tdy_url = "https://www.todayonline.com/" page = requests.get(tdy_url).text soup = BeautifulSoup(page) soup # Returns me a HTML with javascript text soup.find_all('h3') ### Returns me empty list []  Method 2: Selenium + BeautifulSoup tdy_url = "https://www.todayonline.com/" options = Options() options.headless = True driver = webdriver.Chrome("chromedriver",options=options) driver.get(tdy_url) time.sleep(10) html = driver.page_source soup = BeautifulSoup(html) soup.find_all('h3') ### Returns me only less than 1/4 of the 'h3' tags found in the original page source  Please help. I have tried scraping other news websites and it is so much easier. Thank you.

Answer: You can access data via API (check out the Network tab):   For example, import requests url = "https://www.todayonline.com/api/v3/news_feed/7" data = requests.get(url).json()
Question: why does pylint complain about this code block? R1705: Unnecessary "elif" after "return" (no-else-return) def f(a): if a == 1: return 1 elif a == 2: return 2 return 3  To prevent the error, I had to create a temporary variable, which feels less pleasant. def f(a): if a == 1: b = 1 elif a == 2: b = 2 else: b = 3 return b  Solution: def f(a): if a == 1: return 1 if a == 2: return 2 return 3 

Answer: The purpose of an else block is to define code that will not be executed if the condition is true, so execution wouldn't continue on to the next block. However, in your code, the main conditional block has a return statement, meaning execution will leave the function, so there's no need for an else block: all subsequent code after the return will, by definition, not be executed if the condition is true. It's redundant. It can be replaced with a simple if.
Question: I have written a python script which will be installed in as windows service. Below is the code: import datetime import logging from logging.handlers import RotatingFileHandler import os import time from random import randint import win32serviceutil import win32service import win32event import servicemanager import socket def setup_logger(logger_name, log_file, level=logging.ERROR): log_formatter = logging.Formatter('%(asctime)s %(message)s') my_handler = RotatingFileHandler(log_file, maxBytes=100 * 1024 * 1024, backupCount=5) my_handler.setFormatter(log_formatter) my_handler.setLevel(level) l = logging.getLogger(logger_name) l.handlers[:] = [] l.addHandler(my_handler) curr_path = os.getcwd() log_file = "F:\\Projects\\TestService\\logs\\application.log" setup_logger('debug', log_file) log = logging.getLogger('debug') class AppServerSvc(win32serviceutil.ServiceFramework): _svc_name_ = "test_service" _svc_display_name_ = "Test Service" def __init__(self, args): win32serviceutil.ServiceFramework.__init__(self, args) self.hWaitStop = win32event.CreateEvent(None, 0, 0, None) socket.setdefaulttimeout(60) self.isrunning = False def SvcStop(self): self.ReportServiceStatus(win32service.SERVICE_STOP_PENDING) win32event.SetEvent(self.hWaitStop) self.isrunning = False def SvcDoRun(self): servicemanager.LogMsg(servicemanager.EVENTLOG_INFORMATION_TYPE, servicemanager.PYS_SERVICE_STARTED, (self._svc_name_, '')) self.isrunning = True self.main() def main(self): while self.isrunning: log.error("Running {}".format(randint(00, 99))) time.sleep(10) if __name__ == '__main__': win32serviceutil.HandleCommandLine(AppServerSvc)  I have run the command python test_service.py install to install the service and got the correct output Installing service test_service Service installed. When I open services tab I can see my service listed there. When I click on start service I am getting below error:  Can anyone please tell me what is wrong in the code due to which its not starting the service. Please help. Thanks UPDATE: I ran the service in debug mode in cmd and looks like it is working fine. But from the services tab, its not working and showing above error. > python test_service.py debug Debugging service test_service - press Ctrl+C to stop. Info 0x40001002 - The test_service service has started.  When starting the service, it gives same error: > python test_service.py start Starting service test_service Error starting service: The service did not respond to the start or control request in a timely fashion.  Not sure why its not running and in debug mode it runs fine. Please help.

Answer: Anyone facing this issue, just copy pywintypes36.dll from Python36\Lib\site-packages\pywin32_system32 to Python36\Lib\site-packages\win32 Helpful commands:  Install a service: python app.py install  Uninstall a service: python app.py remove  Start a service: python app.py start  Update service: python app.py update
Question: Hello I am training a model with TensorFlow and Keras, and the dataset was downloaded from https://www.microsoft.com/en-us/download/confirmation.aspx?id=54765 This is a zip folder that I split in the following directories: . ├── test │ ├── Cat │ └── Dog └── train ├── Cat └── Dog  Test.cat and test.dog have each folder 1000 jpg photos, and train.cat and traing.dog have each folder 11500 jpg photos. The load is doing with this code: batch_size = 16 # Data augmentation and preprocess train_datagen = ImageDataGenerator(rescale=1./255, shear_range=0.2, zoom_range=0.2, horizontal_flip=True, validation_split=0.20) # set validation split # Train dataset train_generator = train_datagen.flow_from_directory( 'PetImages/train', target_size=(244, 244), batch_size=batch_size, class_mode='binary', subset='training') # set as training data # Validation dataset validation_generator = train_datagen.flow_from_directory( 'PetImages/train', target_size=(244, 244), batch_size=batch_size, class_mode='binary', subset='validation') # set as validation data test_datagen = ImageDataGenerator(rescale=1./255) # Test dataset test_datagen = test_datagen.flow_from_directory( 'PetImages/test')  THe model is training with the following code: history = model.fit(train_generator, validation_data=validation_generator, epochs=5)  And i get the following input: Epoch 1/5 1150/1150 [==============================] - ETA: 0s - loss: 0.0505 - accuracy: 0.9906  But when the epoch is in this point I get the following error:  UnidentifiedImageError: cannot identify image file <_io.BytesIO object at 0x7f9e185347d0>  How can I solve this, in order to finish the training? Thanks

Answer: Try this function to check if the image are all in correct format. import os from PIL import Image folder_path = 'data\img' extensions = [] for fldr in os.listdir(folder_path): sub_folder_path = os.path.join(folder_path, fldr) for filee in os.listdir(sub_folder_path): file_path = os.path.join(sub_folder_path, filee) print('** Path: {} **'.format(file_path), end="\r", flush=True) im = Image.open(file_path) rgb_im = im.convert('RGB') if filee.split('.')[1] not in extensions: extensions.append(filee.split('.')[1])
Question: I was trying to install OpenCV4 in a docker on jetson nano. It has jetpack 4.4 s os. The docker was successfully created and Tensorflow is running but while installing OpenCV using pip it is showing CMake error. root@5abf405fb92d:~# pip3 install opencv-python Collecting opencv-python Downloading opencv-python-4.4.0.42.tar.gz (88.9 MB) |████████████████████████████████| 88.9 MB 2.5 kB/s Installing build dependencies ... done Getting requirements to build wheel ... done Preparing wheel metadata ... done Requirement already satisfied: numpy>=1.13.3 in /usr/local/lib/python3.6/dist-packages (from opencv-python) (1.18.4) Building wheels for collected packages: opencv-python Building wheel for opencv-python (PEP 517) ... error ERROR: Command errored out with exit status 1: command: /usr/bin/python3 /usr/local/lib/python3.6/dist-packages/pip/_vendor/pep517/_in_process.py build_wheel /tmp/tmpqpzwrofy cwd: /tmp/pip-install-93nxibky/opencv-python Complete output (9 lines): File "/tmp/pip-build-env-o_hualnr/overlay/lib/python3.6/site-packages/skbuild/setuptools_wrap.py", line 560, in setup cmkr = cmaker.CMaker(cmake_executable) File "/tmp/pip-build-env-o_hualnr/overlay/lib/python3.6/site-packages/skbuild/cmaker.py", line 95, in __init__ self.cmake_version = get_cmake_version(self.cmake_executable) File "/tmp/pip-build-env-o_hualnr/overlay/lib/python3.6/site-packages/skbuild/cmaker.py", line 82, in get_cmake_version "Problem with the CMake installation, aborting build. CMake executable is %s" % cmake_executable) Traceback (most recent call last): Problem with the CMake installation, aborting build. CMake executable is cmake ---------------------------------------- ERROR: Failed building wheel for opencv-python Failed to build opencv-python ERROR: Could not build wheels for opencv-python which use PEP 517 and cannot be installed directly 

Answer: I had the same problem and i did this, pip install --upgrade pip setuptools wheel  then install opencv again, pip install opencv-python  this worked for me
Question: I have a DataFrame like this: vals = {"operator": [1, 1, 1, 2, 3, 5], "nextval": [2, 3, 6, 4, 5, 6]} df = pd.DataFrame(vals) operator nextval 0 1 2 1 1 3 2 1 6 3 2 4 4 3 5 5 5 6  What I'm trying to do is get a list of all the possible paths from a starting point, like 1, and an ending point, like 6, using the operators and nextvals, not strictly the shortest path. The output can be flexible, but I'm looking for something like this or that communicates this: 1 -> 6 1 -> 2 -> 4 1 -> 3 -> 5 -> 6  I'm able to get it close, but not sure how to get the recursion right since the dict can't handle 2 same keys: import pandas as pd vals = {"operator": [1, 1, 1, 2, 3, 5], "nextval": [2, 3, 6, 4, 5, 6]} df = pd.DataFrame(vals) df1 = df.set_index("operator") dictvals = {} for x in df1.index.unique(): dictvals[x] = [] df2 = df1.loc[x] if isinstance(df2, pd.DataFrame): for idx, rowdata in df2.iterrows(): dictvals[x].append(rowdata["nextval"]) else: dictvals[x] = df2[0] print(dictvals) {1: [2, 3, 6], 2: 4, 3: 5, 5: 6} 

Answer: Check with networkx , you need a direction graph with 'root' to 'leaf' path import networkx as nx G=nx.from_pandas_edgelist(df,source='operator',target='nextval', edge_attr=None, create_using=nx.DiGraph()) road=[] for n in G: if G.out_degree(n)==0: #leaf road.append(nx.shortest_path(G, 1, n)) road Out[82]: [[1, 2, 4], [1, 3, 5, 6]]  Update import networkx as nx G=nx.from_pandas_edgelist(df,source='operator',target='nextval', edge_attr=None, create_using=nx.DiGraph()) road=[] for n in G: if G.out_degree(n)==0: #leaf road.append(list(nx.all_simple_paths(G, 1, n))) road Out[509]: [[[1, 3, 5, 6], [1, 6]], [[1, 2, 4]]]
Question: I currently type hint a function returning tuple as follows: FuncOutput = Tuple[nib.Nifti1Image, nib.Nifti1Image, nib.Nifti1Image, nib.Nifti1Image, nib.Nifti1Image, nib.Nifti1Image, nib.Nifti1Image]  Is there a way to do this in a concise manner where I can specify the length without typing it so many times?

Answer: No. typing.Tuple only supports typing each element or a variable number of elements.
Question: I am trying to setup structlog and set log level. My code looks like this: import structlog import logging filepath=open("out.log",'a') logging.basicConfig( level=logging.INFO ) structlog.configure( processors=[structlog.stdlib.filter_by_level], wrapper_class=structlog.BoundLogger, context_class=dict, logger_factory=structlog.PrintLoggerFactory(filepath), ) logger = structlog.getLogger() logger.info('test')  This fails: Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/usr/local/lib/python3.8/site-packages/structlog/_base.py", line 189, in _proxy_to_logger args, kw = self._process_event(method_name, event, event_kw) File "/usr/local/lib/python3.8/site-packages/structlog/_base.py", line 149, in _process_event event_dict = proc(self._logger, method_name, event_dict) File "/usr/local/lib/python3.8/site-packages/structlog/stdlib.py", line 381, in filter_by_level if logger.isEnabledFor(_NAME_TO_LEVEL[name]): AttributeError: 'PrintLogger' object has no attribute 'isEnabledFor'  Okay, sure. I am not supposed to use PrintLogger with stdlib processors. But I want to filter by log level (because that's how logging usually works, eh?) So how do I do that? I assume I need to use some other logger factory, but which one? Of course structlog.stdlib.LoggerFactory works, but it doesn't redirect to a file. So I said: okay I will create my own filter: def my_filter_by_level(logger, name, event_dict): if True: return event_dict else: raise DropEvent  ... processors=[my_filter_by_level],  And when I try to use the logger I get: Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/usr/local/lib/python3.8/site-packages/structlog/_base.py", line 190, in _proxy_to_logger return getattr(self._logger, method_name)(*args, **kw) TypeError: msg() got an unexpected keyword argument 'organization'  This is coming from logger = logger.bind(**{"organization": "blah"})  but... why? What is wrong with my processor?

Answer: Well, the reason my filter didn't work is this: https://www.structlog.org/en/stable/processors.html#adapting Last filter in chain is special and can't just return a dictionary. And the following filter did the trick: def my_filter_by_level(logger, name, event_dict): this_level = structlog.stdlib._NAME_TO_LEVEL[name] set_level = structlog.stdlib._NAME_TO_LEVEL[loglevel] if this_level >= set_level: return event_dict else: raise structlog.DropEvent
Question: In Python 3.8+, is it possible to check whether a numpy array is being stored in shared memory? In the following example, a numpy array sharedArr was created using the buffer of a multiprocessing.shared_memory.SharedMemory object. Will like to know if we can write a function that can detect whether SharedMemory is used. import numpy as np from multiprocessing import shared_memory if __name__ == '__main__': # Created numpy array `sharedArr`in shared memory arr = np.zeros(5) shm = shared_memory.SharedMemory(create=True, size=arr.nbytes) sharedArr = np.ndarray(arr.shape, dtype=arr.dtype, buffer=shm.buf) sharedArr[:] = arr[:] # How to tell if numpy array is stored in shared memory? print(type(sharedArr)) # <class 'numpy.ndarray'> print(hex(id(sharedArr))) # 0x7fac99469f30 shm.close() shm.unlink() 

Answer: In this particular case, you can use the base attribute of the shared array. The attribute is a reference to the underlying object from which this array derives its memory. This is None for most arrays, to indicate that such an array owns its data. Running this code on my machine indicates that this array's base is a mmap object: >>> sharedArr.base <mmap.mmap at 0x11a4aa670>  If you still have a reference to the shared memory object from which the array was allocated, you can compare the array's base to the shared memory segment's memory map: >>> sharedArr.base is shm._mmap True  If you don't have the shm object lying around, as you wouldn't in a standalone function which could hypothetically perform this task, I doubt there's a portable and foolproof way to do it. Since NumPy provides its own memory-map object, it may suffice for your case to do the former check. That is, make the assumption that if the array is backed by a vanilla, builtin Python memory map, it is allocated from shared memory: import mmap def array_is_from_shared_memory(arr): return isinstance(arr.base, mmap.mmap)  This works in your particular example, but you'd have to be careful with it, clearly document the assumptions that it makes, and test that it provides you with the actual information you need in your exact application.
Question: The project structure my_package ├── my_package │ ├── __init__.py │ └── my_module.py └── setup.py  The module my_module.py has a single func function I am attempting to import. The setup.py file has the following content. from setuptools import setup, find_packages setup( name='my_package', packages=find_packages(where='my_package'), version='1.0' )  The import API I'm installing the package with: virtualenv --python=/usr/bin/python3.8 venv source venv/bin/activate python my_package/setup.py install  To then import it with: import my_package from my_package import my_module  However, the second import fails with: ImportError: cannot import name 'my_module' from 'my_package' (unknown location)  Further more, running dir(my_package) reveals that indeed the my_module name did not get imported. ['__doc__', '__file__', '__loader__', '__name__', '__package__', '__path__', '__spec__'] Similar questions on SO setup.py installed package can't be imported provided solution proved non sucessfull. ImportError: cannot import name 'Serial' from 'serial' (unknown location) adding a __init__.py file in my_package/my_package didn't work. Git repo I've placed an example of the issue at GitLab

Answer: You're running your test.py script in the parent directory of your my_package directory. As a result, test.py will try and import the my_package subdirectory as a package/module, not your installed package. You will need to move to a directory that doesn't contain your source code and then run test. This could be as simply as running it inside a subdirectory test in the my_package main directory. Just make sure you cd into that directory explicitly, instead of running it with a full path (like, for example, python3.8 my_package/test/test.py, because then it will still import the wrong my_package. The reason for this (and cause of your problem) is that Python automatically includes the current working directory in your sys.path, at the start, and will thus try and import the main my_package directory as a package.
Question: I've started learning Python recently and I don't understand why Python behaves like this:  >>> "OK" 'OK' >>> """OK""" 'OK' >>> "not Ok' File "<stdin>", line 1 "not Ok' ^ SyntaxError: EOL while scanning string literal >>> "not OK""" 'not OK'  Why doesn't it give an error for the last statement as the number of quotes does not match?

Answer: The final """ is not recognized as a triple-quotation, but a single " (to close the current string literal) followed by an empty string ""; the two juxtaposed string literals are concatenated. The same behavior can be more readily recognized by putting a space between the closing and opening ". >>> "not OK" "" 'not OK'
Question: There is a big space between the border/x-axe of the graph and the first and last bar in a bar plot in pyplot (red arrows in the first picture). In the image below, it looks fine in the graph on the left, but it's wasting a lot of space in the graph on the right. The larger the graph, the larger the space.  See how much space is wasted in the second picture:  Any idea how to fix that? The code: plt.figure(figsize=figsize) grid = plt.GridSpec(figsize[1], figsize[0], wspace=120/figsize[0]) plt.suptitle(feature.upper()) # NaN plot plt.subplot(grid[:, :2]) plt.title('Présence') plt.ylabel('occurences') plt.bar([0, 1], [df.shape[0] - nan, nan], color=['#4caf50', '#f44336']) plt.xticks([0, 1], ['Renseigné', 'Absent'], rotation='vertical') # Distrib plot plt.subplot(grid[:, 2:]) plt.title('Distribution') x_pos = [i for i, _ in enumerate(sizes)] plt.bar(x_pos, sizes) plt.xticks(x_pos, labels, rotation='vertical') plt.show()  df is my pandas DataFrame, nan the amount of null value for the feature I'm plotting, sizes is the number of occurrences for each value, and labels are the corresponding label's value.

Answer: You can use plt.margins(x=0, tight=True). The default margins are 0.05 which means that 5% of the distance between the first and last x-values is used as a margin. Choosing a small value such as plt.margins(x=0-01, tight=True) would leave a bit of margin, so the bars don't look glued to the axes. In some situations wider margins are desired. For example when drawing only two very narrow bars. So, the exact value for a plot to look nice depends on multiple factors as well as on personal preferences. Note that plt.margins is best called after the last function that adds elements to the plot. Also note that plt.xlim has a similar function of changing the drawing limits. Here is an example that sets xlim to have the same distance between the axes and the bars as between the bars themselves. import matplotlib.pyplot as plt fig, axes = plt.subplots(ncols=3, figsize=(12, 3)) for ax in axes: width = 0.4 ax.bar([0, 1, 2], [2, 3, 5], width=width, color='turquoise') if ax == axes[0]: ax.margins(x=0.01) ax.set_title('1% margins') elif ax == axes[1]: ax.margins(x=0.1) ax.set_title('10% margins') else: ax.set_xlim(-1+width/2, 3-width/2) ax.set_title('setting xlim') plt.show()
Question: I have a time-range and an interval, I need to split the time range into multiple time periods based on interval value. For example, time range is 9:30 to 11:30 and the interval is 30, the output time periods should be in a list as datetime objects Output: [ 2020-08-24 9:30 - 2020-08-24 10:00, 2020-08-24 10:00 - 2020-08-24 10:30 2020-08-24 10:30 - 2020-08-24 11:00, 2020-08-24 11:00 - 2020-08-24 11:30 ] 

Answer: You can do arithmetic on datetime objects by adding timedelta objects. You probably need to decide exactly what behaviour is required if the interval per period is not an exact divisor of the total, but this example would give a final short period in that case. import datetime tstart = datetime.datetime(2020,8,24,9,30) tend = datetime.datetime(2020,8,24,11,30) interval = datetime.timedelta(minutes=30) periods = [] period_start = tstart while period_start < tend: period_end = min(period_start + interval, tend) periods.append((period_start, period_end)) period_start = period_end print(periods)  This gives (with newlines inserted for readability): [(datetime.datetime(2020, 8, 24, 9, 30), datetime.datetime(2020, 8, 24, 10, 0)), (datetime.datetime(2020, 8, 24, 10, 0), datetime.datetime(2020, 8, 24, 10, 30)), (datetime.datetime(2020, 8, 24, 10, 30), datetime.datetime(2020, 8, 24, 11, 0)), (datetime.datetime(2020, 8, 24, 11, 0), datetime.datetime(2020, 8, 24, 11, 30))]  For the string output format that you want, you could do something like this: def format_time(dt): return dt.strftime("%Y-%m-%d %H:%M") print(['{} - {}'.format(format_time(start), format_time(end)) for start, end in periods])  to give: ['2020-08-24 09:30 - 2020-08-24 10:00', '2020-08-24 10:00 - 2020-08-24 10:30', '2020-08-24 10:30 - 2020-08-24 11:00', '2020-08-24 11:00 - 2020-08-24 11:30']