eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: In Keras, why is it that input_shape does not include the batch dimension when passed as an argument to layers like Dense but DOES include the batch dimension when input_shape is passed to the build method of a model? import tensorflow as tf from tensorflow.keras.layers import Dense if __name__ == "__main__": model1 = tf.keras.Sequential([Dense(1, input_shape=[10])]) model1.summary() model2 = tf.keras.Sequential([Dense(1)]) model2.build(input_shape=[None, 10]) # why [None, 10] and not [10]? model2.summary()  Is this a conscious choice of API design? If it is, why?

Answer: You can specify the input shape of your model in several different ways. For example by providing one of the following arguments to the first layer of your model:  batch_input_shape: A tuple where the first dimension is the batch size. input_shape: A tuple that does not include the batch size, e.g., the batch size is assumed to be None or batch_size, if specified. input_dim: A scalar indicating the dimension of the input.  In all these cases, Keras is internally storing an attribute _batch_input_size to build the model. Regarding the build method, my guess is that this is indeed a conscious choice - information about the batch size might be useful to build the model in some (perhaps unthought-of) situations. Therefore, a framework that includes the batch dimension as input to build is more generic and complete than a framework that doesn't. Nonetheless, I agree with you that naming the argument batch_input_shape instead of input_shape would make everything more consistent.  It is also worth mentioning that users rarely need to call the build method by themselves. This happens internally when it is needed. Nowadays, it is even possible to ignore the input_shape argument when creating the model (although methods like summary will then not work until the model is built). In this case, Keras is able to infer the input shape from the argument x of fit.
Question: I am writing a REST API using aiohttp. Some of the coroutines need to call the database using the aiomysql library. The documentation for aiomysql has the following example: import asyncio import aiomysql loop = asyncio.get_event_loop() async def test_example(): conn = await aiomysql.connect(host='127.0.0.1', port=3306, user='root', password='', db='mysql', loop=loop) cur = await conn.cursor() await cur.execute("SELECT Host,User FROM user") print(cur.description) r = await cur.fetchall() print(r) await cur.close() conn.close() loop.run_until_complete(test_example())  My question is concerning the definition of the global variable loop: loop = asyncio.get_event_loop()  Do I really need to keep loop as a global variable somewhere, or can I just call asyncio.get_event_loop() when I need it? For example, it the code example above, I could get the event loop when I am connecting to the database:  conn = await aiomysql.connect(host='127.0.0.1', port=3306, user='root', password='', db='mysql', loop=asyncio.get_event_loop())  Is there a non-trivial runtime cost to calling asyncio.get_event_loop() or something else that I am missing?

Answer: loop argument should go. aiomysql should be updated to don't accept the loop. You can just skip loop=... in your code right now because aiomysql.connect() has the default loop=None value for the argument. In general, asyncio.get_event_loop() will be deprecated; asyncio.get_running_loop() is recommended for the usage from an async code when needed. Passing an explicit loop to asyncio API is deprecated starting from Python 3.8.
Question: Currently I have something like this: @dataclass(frozen=True) class MyClass: a: str b: str c: str d: Dict[str, str]  ...which is all well and good except dicts are mutable, so I can't use my class to key another dictionary. Instead, I'd like field d to be something like a FrozenSet[Tuple[str, str]], but I'd still like someone constructing an instance of my class to be able to pass a dictionary on the constructor as this is much more intuitive. So I'd like to do something like @dataclass(frozen=True) class MyClass: a: str b: str c: str d: FrozenSet[Tuple[str, str]] = field(init=False) def __init__(self, a, b, c, d: Dict[str, str]): self.original_generated_init(a, b, c) # ??? object.setattr(self, 'd', frozenset(d.items())) # required because my dataclass is frozen  How do I achieve this? Alternatively is there a more elegant way to achieve the same thing?

Answer: You can use an InitVar and assign to d in __post_init__: @dataclass(frozen=True) class MyClass: a: str b: str c: str d: FrozenSet[Tuple[str, str]] = field(init=False) d_init: InitVar[Dict[str, str]] def __post_init__(self, d_init): object.__setattr__(self, 'd', frozenset(d_init.items()))
Question: I want to click on this button when <Span text is "OK": <button type="button" class="ant-btn ant-btn-primary" style="float: right;"><span>OK</span></button>  The soluton would be something like that (this code don´t work because it´s not clickable): driver.find_element_by_xpath('//span[text()="OK"]').click()  I found a way that works, but I can´t garantee the text inside Span is "OK": button = driver.find_element_by_css_selector('#root > section > section > div.ant-row > div:nth-child(2) > div.ant-spin-nested-loading > div > div > div > div > div.ant-collapse-content.ant-collapse-content-active > div > div > div > div:nth-child(17) > button.ant-btn.ant-btn-primary') button.click()  How can I click on buttons based on Span tag? Edit: Button code line

Answer: So, based on your button HTML <button type="button" class="ant-btn ant-btn-primary" style="float: right;"><span>OK</span></button>  I think that the following xapth would help you. //button[contains(@class, 'ant-btn-primary')]//*[contains(., 'OK')]/..  The /.. helps you traverse back 1 level; sending you to the main button. In order to search for the "OK" text in a span, you need to use the node / period search. Which is why I did //*[contains(., 'OK')]. To find this button and then click on it, you would do the following driver.find_element(By.XPATH, "//button[contains(@class, 'ant-btn-primary')]//*[contains(., 'OK')]/..").click()  ALTERNATE XPATH //button[contains(@class, 'ant-btn-primary') and contains(., 'OK')]  And you would click on the button as such driver.find_element(By.XPATH, "//button[contains(@class, 'ant-btn-primary') and contains(., 'OK')]").click()
Question: I want to implement a way to repeat a section of code as many times as it's needed using a context manager only, because of its pretty syntax. Like this: with try_until_success(attempts=10): command1() command2() command3()  The commands must be executed once if no errors happen. And they should be executed again if an error occurred, until 10 attempts has passed, if so the error must be raised. For example, it can be useful to reconnect to a data base. The syntax I represented is literal, I do not want to modify it (so do not suggest me to replace it with a kind of for of while statements). Is there a way to implement try_until_success in Python to do what I want? What I tried is: from contextlib import contextmanager @contextmanager def try_until_success(attempts=None): counter = 0 while True: try: yield except Exception as exc: pass else: break counter += 1 if attempts is not None and counter >= attempts: raise exc  And this gives me the error: RuntimeError: generator didn't stop after throw()  I know, there are many ways to reach what I need using a loop instead of with-statement or with the help of a decorator. But both have syntax disadvantages. For example, in case of a loop I have to insert try-except block, and in case of a decorator I have to define a new function. I have already looked at the questions: How do I make a contextmanager with a loop inside? Conditionally skipping the body of Python With statement They did not help in my question.

Answer: The problem is that the body of the with statement does not run within the call to try_until_success. That function returns an object with a __enter__ method; that __enter__ method calls and returns, then the body of the with statement is executed. There is no provision for wrapping the body in any kind of loop that would allow it to be repeated once the end of the with statement is reached.
Question: Halfway through Architecture Patterns with Python, I have two questions about how should the Domain Model Classes be structured and instantiated. Assume on my Domain Model I have the class DepthMap: class DepthMap: def __init__(self, map: np.ndarray): self.map = map  According to what I understood from the book, this class is not correct since it depends on Numpy, and it should depend only on Python primitives, hence the question: Should Domain Model classes rely only on Python primitives, or is there an exception? Assuming the answer to the previous question is that classes should solely depend on primitives, what would the correct way create a DepthMap from a Numpy array be? Assume now I have more formats from where I can make a DepthMap object. class DepthMap: def __init__(self, map: List): self.map = map @classmethod def from_numpy(cls, map: np.ndarray): return cls(map.tolist()) @classmethod def from_str(cls, map: str): return cls([float(i) for i in s.split(',')])  or a factory: class DepthMapFactory: @staticmethod def from_numpy(map: np.ndarray): return DepthMap(map.tolist()) @staticmethod def from_str(map: str): return DepthMap([float(i) for i in s.split(',')])  I think even the Repository Pattern, which they go through in the book, could fit in here: class StrRepository: def get(map: str): return DepthMap([float(i) for i in s.split(',')]) class NumpyRepository: def get(map: np.ndarray): return DepthMap(map.tolist())  The second question: When creating a Domain Model Object from different sources, what is the correct approach? Note: My background is not software; hence some OOP concepts may be incorrect. Instead of downvoting, please comment and let me know how to improve the question.

Answer: I wrote the book, so I can at least have a go at answering your question. You can use things other than primitives (str, int, boolean etc) in your domain model. Generally, although we couldn't show it in the book, your model classes will contain whole hierarchies of objects. What you want to avoid is your technical implementation leaking into your code in a way that makes it hard to express your intent. It would probably be inappropriate to pass instances of Numpy arrays around your codebase, unless your domain is Numpy. We're trying to make code easier to read and test by separating the interesting stuff from the glue. To that end, it's fine for you to have a DepthMap class that exposes some behaviour, and happens to have a Numpy array as its internal storage. That's not any different to you using any other data structure from a library. If you've got data as a flat file or something, and there is complex logic involved in creating the Numpy array, then I think a Factory is appropriate. That way you can keep the boring, ugly code for producing a DepthMap at the edge of your system, and out of your model. If creating a DepthMap from a string is really a one-liner, then a classmethod is probably better because it's easier to find and understand.
Question: When using dns.resolver.Resolver() it returns a warning that I should use dns.resolver.Resolver.resolve () When I changed it, it further gives error: TypeError: resolve() missing 2 required positional arguments: 'self' and 'qname'  This is the code:  # my_resolver = dns.resolver.Resolver() my_resolver = dns.resolver.Resolver.resolve() answers = my_resolver.query(host, "A") answer_txt = my_resolver.query(host, "TXT")  Any suggestions to fix this?

Answer: faced the same problem You should remove .resolve from my_resolver = dns.resolver.Resolver.resolve() and replace my_resolver.query() with my_resolver.resolve() Example:  my_resolver = dns.resolver.Resolver() answers = my_resolver.resolve(host, "A") answer_txt = my_resolver.resolve(host, "TXT")
Question: I need to push data to a website through their API and I'm trying to find a way to use F-string to pass the variable in the data list but couldn't find a way. Here's what I've tried so far: today = datetime.date.today() tomorrow = today + datetime.timedelta(days = 1) #trying to pass *tomorrow* value with f-string below data = f'[{"date": "{tomorrow}", "price": {"amount": 15100}}]' response = requests.put('https://api.platform.io/calendar/28528204', headers=headers, data=data)  How can I achieve this?

Answer: I would personally do something like this: import json import datetime today = datetime.date.today() tomorrow = today + datetime.timedelta(days = 1) data = [{ "date": str(tomorrow), "price": { "amount": 15100 } }] print(json.dumps(data))  Of course, after this, you can do anything you want with json.dumps(data): in your case, send it in a request.
Question: This is quite simple case, but I did not find any easy way to do it so far. The idea is to get a set of distances between all the points defined in a GeoDataFrame and the ones defined in another GeoDataFrame. import geopandas as gpd import pandas as pd # random coordinates gdf_1 = gpd.GeoDataFrame(geometry=gpd.points_from_xy([0, 0, 0], [0, 90, 120])) gdf_2 = gpd.GeoDataFrame(geometry=gpd.points_from_xy([0, 0], [0, -90])) print(gdf_1) print(gdf_2) # distances are calculated elementwise print(gdf_1.distance(gdf_2))  This produces the element-wise distance between points in gdf_1 and gdf_2 that share the same index (with also a warning because the two GeoSeries do not have the same index, which will be my case).  geometry 0 POINT (0.000 0.000) 1 POINT (0.000 90.000) 2 POINT (0.000 120.000) geometry 0 POINT (0.00000 0.00000) 1 POINT (0.00000 -90.00000) /home/seydoux/anaconda3/envs/chelyabinsk/lib/python3.8/site-packages/geopandas/base.py:39: UserWarning: The indices of the two GeoSeries are different. warn("The indices of the two GeoSeries are different.") 0 0.0 1 180.0 2 NaN  The question is; how is it possible to get a series of all points to points distances (or at least, the unique combinations of the index of gdf_1 and gdf_2 since it is symmetric). EDIT  In this post, the solution is given for a couple of points; but I cannot find a straightforward way to combine all points in two datasets.  In this post only element-wise operations are proposed.  An analogous question was also raised on the GitHub repo of geopandas. One of the proposed solution is to use the apply method, without any detailed answer.  

Answer: You have to apply over each geometry in first gdf to get distance to all geometric in second gdf. import geopandas as gpd import pandas as pd # random coordinates gdf_1 = gpd.GeoDataFrame(geometry=gpd.points_from_xy([0, 0, 0], [0, 90, 120])) gdf_2 = gpd.GeoDataFrame(geometry=gpd.points_from_xy([0, 0], [0, -90])) gdf_1.geometry.apply(lambda g: gdf_2.distance(g))   0 1 0 0.0 90.0 1 90.0 180.0 2 120.0 210.0
Question: I've been using ggplot for a long time now and am very comfortable using it in R. I am working in Python right now for school and I am having the toughest time understanding this error. When I try to use scale_color_manual to manually assign colors to my variable called "CellTypeOther" which has unique values 0/1, I keep getting the following error: NameError: name 'c' is not defined  Here is my code to create the ggplot: from plotnine import * plot = ( ggplot(Y_tsne,aes(x = 'X',y = 'Y')) + geom_point(aes(color = 'CellTypeOther'),alpha = 0.4,size = 2) + scale_color_manual(c("blue","red")) )  The plot renders fine without the last line. Does anyone have any clue as to what might be going on? I can't share my data because it is sensitive. Important note: I am using the plotnine Python module to utilize ggplot2.

Answer: You should use [] rather than c() in python:
Question: I'm using FastAPI to receive an image, process it and then return the image as a FileResponse. But the returned file is a temporary one that need to be deleted after the endpoint return it. @app.post("/send") async def send(imagem_base64: str = Form(...)): # Convert to a Pillow image image = base64_to_image(imagem_base64) temp_file = tempfile.mkstemp(suffix = '.jpeg') image.save(temp_file, dpi=(600, 600), format='JPEG', subsampling=0, quality=85) return FileResponse(temp_file) # I need to remove my file after return it os.remove(temp_file)  How can I delete the file after return it ?

Answer: You can delete a file in a background task, as it will run after the response is sent. import os import tempfile from fastapi import FastAPI from fastapi.responses import FileResponse from starlette.background import BackgroundTasks app = FastAPI() def remove_file(path: str) -> None: os.unlink(path) @app.post("/send") async def send(background_tasks: BackgroundTasks): fd, path = tempfile.mkstemp(suffix='.txt') with os.fdopen(fd, 'w') as f: f.write('TEST\n') background_tasks.add_task(remove_file, path) return FileResponse(path)  Another approach is to use dependency with yield. The finally block code will be executed after the response is sent and even after all background tasks have been completed. import os import tempfile from fastapi import FastAPI, Depends from fastapi.responses import FileResponse app = FastAPI() def create_temp_file(): fd, path = tempfile.mkstemp(suffix='.txt') with os.fdopen(fd, 'w') as f: f.write('TEST\n') try: yield path finally: os.unlink(path) @app.post("/send") async def send(file_path=Depends(create_temp_file)): return FileResponse(file_path)  Note: mkstemp() returns a tuple with a file descriptor and a path.
Question: I need to find a longest path in an unweighted graph from s to t. I am using NetworkX, which has an algorithm for finding a longest path in a directed, acyclic graph, but I am not able to specify the source and the target nodes. I have not been able to find any info online, but it seems like such an obvious algorithm to have laying around. Is there any way I can do this?

Answer: "[T]he longest path problem is the problem of finding a simple path of maximum length in a given graph."[1] NetworkX has a simple_paths module, that contains the function all_simple_paths. Below is one solution for finding the longest simple paths between two nodes. from typing import List import networkx as nx def longest_simple_paths(graph, source, target) -> List[List]: longest_paths = [] longest_path_length = 0 for path in nx.all_simple_paths(G, source=source, target=target): if len(path) > longest_path_length: longest_path_length = len(path) longest_paths.clear() longest_paths.append(path) elif len(path) == longest_path_length: longest_paths.append(path) return longest_paths G = nx.complete_graph(4) longest_paths = longest_simple_paths(G, source=0, target=3) if longest_paths: print(f"Longest simple path contains {len(longest_paths[0])} nodes") print(longest_paths)  Output Longest simple path contains 4 nodes [[0, 1, 2, 3], [0, 2, 1, 3]]   [1] Wikipedia contributors. "Longest path problem." Wikipedia, The Free Encyclopedia. Available from: https://en.wikipedia.org/wiki/Longest_path_problem. Accessed 8 November 2020.
Question: The below question is for people who use PyCharm. There are nested for loops and tqdm is used for progress bars corresponding to each for loop. The code is shown below. from tqdm import tqdm import time for i in tqdm(range(5), desc="i", colour='green'): for j in tqdm(range(10), desc="j", colour='red'): time.sleep(0.5)  But the problem is, the progress bars for the inner loop appear in newline everytime there is an update in the bar as shown below. i: 0%| | 0/5 [00:00<?, ?it/s] j: 0%| | 0/10 [00:00<?, ?it/s] j: 10%|█ | 1/10 [00:00<00:04, 1.94it/s] j: 20%|██ | 2/10 [00:01<00:04, 1.94it/s] j: 30%|███ | 3/10 [00:01<00:03, 1.96it/s] j: 40%|████ | 4/10 [00:02<00:03, 1.96it/s] j: 50%|█████ | 5/10 [00:02<00:02, 1.97it/s] j: 60%|██████ | 6/10 [00:03<00:02, 1.97it/s] j: 70%|███████ | 7/10 [00:03<00:01, 1.97it/s] j: 80%|████████ | 8/10 [00:04<00:01, 1.98it/s] j: 90%|█████████ | 9/10 [00:04<00:00, 1.98it/s] j: 100%|██████████| 10/10 [00:05<00:00, 1.98it/s] i: 20%|██ | 1/5 [00:05<00:20, 5.06s/it] j: 0%| | 0/10 [00:00<?, ?it/s] j: 10%|█ | 1/10 [00:00<00:04, 2.00it/s] j: 20%|██ | 2/10 [00:01<00:04, 1.99it/s] j: 30%|███ | 3/10 [00:01<00:03, 1.99it/s] j: 40%|████ | 4/10 [00:02<00:03, 1.99it/s] j: 50%|█████ | 5/10 [00:02<00:02, 1.99it/s] j: 60%|██████ | 6/10 [00:03<00:02, 1.99it/s] j: 70%|███████ | 7/10 [00:03<00:01, 1.99it/s] j: 80%|████████ | 8/10 [00:04<00:01, 1.99it/s] j: 90%|█████████ | 9/10 [00:04<00:00, 1.99it/s] j: 100%|██████████| 10/10 [00:05<00:00, 1.99it/s] i: 40%|████ | 2/5 [00:10<00:15, 5.05s/it]  Setting the parameter 'position` for each loop also doesn't fix the issue. from tqdm import tqdm import time for i in tqdm(range(5), desc="i", colour='green', position=0): for j in tqdm(range(10), desc="j", colour='red', position=1): time.sleep(0.5)  How does one get the progress bar to update in the same line?

Answer: The solution is two fold.  Go to "Edit configurations". Click on the run/debug configuration that is being used. There should be an option "Emulate terminal in output console". Check that. Image added for reference.   Along with the position argument also set the leave argument. The code should look like this. I have added ncols so that the progress bar doesn't take up whole of the console.   from tqdm import tqdm import time for i in tqdm(range(5), position=0, desc="i", leave=False, colour='green', ncols=80): for j in tqdm(range(10), position=1, desc="j", leave=False, colour='red', ncols=80): time.sleep(0.5)  When the code is now run, the output of the console is as shown below. i: 20%|████████▍ | 1/5 [00:05<00:20, 5.10s/it] j: 60%|████████████████████████▌ | 6/10 [00:03<00:02, 1.95it/s]  The updates happen on the same line.
Question: Hi I need to align some time series data with nearest timestamps, so I think pandas.merge_asof could be a good candidate. However, it does not have an option to set how='outer' like in the standard merge method. An example can be: df1:  Value1 Time 2020-07-17 14:25:03.535906075 108 2020-07-17 14:25:05.457247019 110 2020-07-17 14:25:07.467777014 126  df2:  Value2 Time 2020-07-17 14:25:03.535018921 222 2020-07-17 14:25:04.545104980 150 2020-07-17 14:25:07.476825953 60  Then for example, do this merge_asof: pd.merge_asof(df1, df2, left_index=True, right_index=True, direction='nearest', tolerance=pd.Timedelta('0.3s'))  The results will be:  Value1 Value2 Time 2020-07-17 14:25:03.535906075 108 222.0 2020-07-17 14:25:05.457247019 110 NaN 2020-07-17 14:25:07.467777014 126 60.0  But what I want is:  Value1 Value2 Time 2020-07-17 14:25:03.535906075 108 222.0 2020-07-17 14:25:04.545104980 NaN 150.0 <---- this is the difference 2020-07-17 14:25:05.457247019 110 NaN 2020-07-17 14:25:07.467777014 126 60.0  basically like a full outer join. Any suggestion? Thanks in advance. EDIT: So this is the case with 2 dataframes. What if, say for example, there are 10 dataframes (i.e. df1, df2, ..., df10) need to do this "nearest" merging, what will be a good way to do?

Answer: This seems simple enough yet no direct solution. There's an option to merge again to bring in the missing rows: # enumerate the rows of `df2` to later identify which are missing df2 = df2.reset_index().assign(idx=np.arange(df2.shape[0])) (pd.merge_asof(df1.reset_index(), df2[['Time','idx']], on='Time', direction='nearest', tolerance=pd.Timedelta('0.3s')) .merge(df2, on='idx', how='outer') # merge back on row number .assign(Time=lambda x: x['Time_x'].fillna(x['Time_y'])) # fill the time .set_index(['Time']) # set index back .drop(['Time_x','Time_y','idx'], axis=1) .sort_index() ) Value1 Value2 Time 2020-07-17 14:25:03.535906075 108.0 222.0 2020-07-17 14:25:04.545104980 NaN 150.0 2020-07-17 14:25:05.457247019 110.0 NaN 2020-07-17 14:25:07.467777014 126.0 60.0