eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I have problem with sns lineplot and scatterplot. Basically what I'm trying to do is to connect dots of a scatterplot to present closest line joining mapped points. Somehow lineplot is changing width when facing points with tha same x axis values. I want to lineplot to be same, solid line all the way. The code: import pandas as pd import numpy as np import matplotlib.pyplot as plt import seaborn as sns %matplotlib inline data = {'X': [13, 13, 13, 12, 11], 'Y':[14, 11, 13, 15, 20], 'NumberOfPlanets':[2, 5, 2, 1, 2]} cts = pd.DataFrame(data=data) plt.figure(figsize=(10,10)) sns.scatterplot(data=cts, x='X', y='Y', size='NumberOfPlanets', sizes=(50,500), legend=False) sns.lineplot(data=cts, x='X', y='Y',estimator='max', color='red') plt.show()  The outcome:  Any ideas? EDIT: If I try using pyplot it doesn't work either: Code: plt.plot(cts['X'], cts['Y'])  Outcome:  I need one line, which connects closest points (basically what is presented on image one but with same solid line).

Answer: Ok, I have finally figured it out. The reason lineplot was so messy is because data was not properly sorted. When I sorted dataframe data by 'Y' values, the outcome was satisfactory. data = {'X': [13, 13, 13, 12, 11], 'Y':[14, 11, 13, 15, 20], 'NumberOfPlanets':[2, 5, 2, 1, 2]} cts = pd.DataFrame(data=data) cts = cts.sort_values('Y') plt.figure(figsize=(10,10)) plt.scatter(cts['X'], cts['Y'], zorder=1) plt.plot(cts['X'], cts['Y'], zorder=2) plt.show()  Now it works. Tested it also on other similar scatter points. Everything is fine :) Thanks!
Question: How can I change the x and y-axis labels in plotly because in matplotlib, I can simply use plt.xlabel but I am unable to do that in plotly. By using this code in a dataframe: Date = df[df.Country=="India"].Date New_cases = df[df.Country=="India"]['7day_rolling_avg'] px.line(df,x=Date, y=New_cases, title="India Daily New Covid Cases")  I get this output:  In this X and Y axis are labeled as X and Y how can I change the name of X and Y axis to "Date" and "Cases"

Answer:  simple case of setting axis title  update_layout( xaxis_title="Date", yaxis_title="7 day avg" )  full code as MWE import pandas as pd import io, requests df = pd.read_csv( io.StringIO( requests.get( "https://raw.githubusercontent.com/owid/covid-19-data/master/public/data/vaccinations/vaccinations.csv" ).text ) ) df["Date"] = pd.to_datetime(df["date"]) df["Country"] = df["location"] df["7day_rolling_avg"] = df["daily_people_vaccinated_per_hundred"] Date = df[df.Country == "India"].Date New_cases = df[df.Country == "India"]["7day_rolling_avg"] px.line(df, x=Date, y=New_cases, title="India Daily New Covid Cases").update_layout( xaxis_title="Date", yaxis_title="7 day avg" )
Question: Reading a large file from S3 ( >5GB) into lambda with the following code: import json import boto3 s3 = boto3.client('s3') def lambda_handler(event, context): response = s3.get_object( Bucket="my-bucket", Key="my-key" ) text_bytes = response['Body'].read() ... return { 'statusCode': 200, 'body': json.dumps('Hello from Lambda!') }  However I get the following error: "errorMessage": "signed integer is greater than maximum" "errorType": "OverflowError" "stackTrace": [ " File \"/var/task/lambda_function.py\", line 13, in lambda_handler\n text_bytes = response['Body'].read()\n" " File \"/var/runtime/botocore/response.py\", line 77, in read\n chunk = self._raw_stream.read(amt)\n" " File \"/var/runtime/urllib3/response.py\", line 515, in read\n data = self._fp.read() if not fp_closed else b\"\"\n" " File \"/var/lang/lib/python3.8/http/client.py\", line 472, in read\n s = self._safe_read(self.length)\n" " File \"/var/lang/lib/python3.8/http/client.py\", line 613, in _safe_read\n data = self.fp.read(amt)\n" " File \"/var/lang/lib/python3.8/socket.py\", line 669, in readinto\n return self._sock.recv_into(b)\n" " File \"/var/lang/lib/python3.8/ssl.py\", line 1241, in recv_into\n return self.read(nbytes, buffer)\n" " File \"/var/lang/lib/python3.8/ssl.py\", line 1099, in read\n return self._sslobj.read(len, buffer)\n" ]  I am using Python 3.8, and I found here an issue with Python 3.8/9 that might be why: https://bugs.python.org/issue42853 Is there any way around this?

Answer: As mentioned in the bug you linked to, the core issue in Python 3.8 is the bug with reading more than 1gb at a time. You can use a variant of the workaround suggested in the bug to read the file in chunks. import boto3 s3 = boto3.client('s3') def lambda_handler(event, context): response = s3.get_object( Bucket="-example-bucket-", Key="path/to/key.dat" ) buf = bytearray(response['ContentLength']) view = memoryview(buf) pos = 0 while True: chunk = response['Body'].read(67108864) if len(chunk) == 0: break view[pos:pos+len(chunk)] = chunk pos += len(chunk) return { 'statusCode': 200, 'body': json.dumps('Hello from Lambda!') }  At best, however, you're going to spend a minute or more of each Lambda run just reading from S3. It would be much better if you could store the file in EFS and read it from there in the Lambda, or use another solution like ECS to avoid reading from a remote data source.
Question: I can use this special escape sequence to print a hyperlink in bash: echo -e '\e]8;;http://example.com\e\\This is a link\e]8;;\e\\\n'  Result (Link I can click on): This is a link  Now I want to generate this in Python: print('\e]8;;http://example.com\e\\This is a link\e]8;;\e\\\n') \e]8;;http://example.com\e\This is a link\e]8;;\e\ print(r'\e]8;;http://example.com\e\\This is a link\e]8;;\e\\\n') \e]8;;http://example.com\e\\This is a link\e]8;;\e\\\n  As you can see, the escape sequence is not interpreted by the shell. Other escape sequences, like the one for bold text, work: print('\033[1mYOUR_STRING\033[0m') YOUR_STRING # <- is actually bold  How can I get Python to format the URL correctly?

Answer: From This answer, after some tries: print('\x1b]8;;' + 'http://example.com' + '\x1b\\' + 'This is a link' + '\x1b]8;;\x1b\\\n' )  Then better: print( '\x1b]8;;%s\x1b\\%s\x1b]8;;\x1b\\' % ( 'http://example.com' , 'This is a link' ) )
Question: Is there a way in C++ of emulating this python syntax a,b = b,(a+b)  I understand this is trivially possible with a temporary variable but am curious if it is possible without using one?

Answer: You can use the standard C++ function std::exchange like #include <utility> //... a = std::exchange( b, a + b );  Here is a demonstration program #include <iostream> #include <utility> int main() { int a = 1; int b = 2; std::cout << "a = " << a << '\n'; std::cout << "b = " << b << '\n'; a = std::exchange( b, a + b ); std::cout << "a = " << a << '\n'; std::cout << "b = " << b << '\n'; }  The program output is a = 1 b = 2 a = 2 b = 3  You can use this approach in a function that calculates Fibonacci numbers.
Question: For example if the list is: [2,1,2,5,7,6,9] there's 3 possible ways of splitting: [2,1,2] [5,7,6,9] [2,1,2,5] [7,6,9] [2,1,2,5,7,6] [9] I'm supposed to calculate how many times the list can be split in a way that every element on the left is smaller than every element on the right. So with this list, the output would be 3. Here's my current solution: def count(t): c= 0 for i in range(len(t)): try: if max(t[:i]) < min(t[i:]): c+=1 except: continue return c  The above code does the right thing, but it's not of O(n) time complexity. How could I achieve the same result, but faster?

Answer: Here's my final answer: def count(t): c = 0 maxx = max(t) right = [0]*len(t) left = [0]*len(t) maxx = t[0] for i in range(0, len(t)): if maxx >= t[i]: left[i] = maxx if maxx < t[i]: maxx = t[i] left[i] = maxx minn = t[-1] for i in range(len(t)-1,-1,-1): if minn <= t[i]: right[i] = minn if minn > t[i]: minn = t[i] right[i] = minn for i in range(0, len(t)-1): if left[i] < right[i+1] : c += 1 return c
Question: I have a spark df with the following schema:  |-- col1 : string |-- col2 : string |-- customer: struct | |-- smt: string | |-- attributes: array (nullable = true) | | |-- element: struct | | | |-- key: string | | | |-- value: string  df: #+-------+-------+---------------------------------------------------------------------------+ #|col1 |col2 |customer | #+-------+-------+---------------------------------------------------------------------------+ #|col1_XX|col2_XX|"attributes":[[{"key": "A", "value": "123"},{"key": "B", "value": "456"}] | #+-------+-------+---------------------------------------------------------------------------+  and the json input for the array look like this: ... "attributes": [ { "key": "A", "value": "123" }, { "key": "B", "value": "456" } ],  I would like to loop attributes array and get the element with key="B" and then select the corresponding value. I don't want to use explode because I would like to avoid join dataframes. Is it possible to perform this kind of operation directly using spark 'Column' ? Expected output will be: #+-------+-------+-----+ #|col1 |col2 |B | | #+-------+-------+-----+ #|col1_XX|col2_XX|456 | #+-------+-------+-----+  any help would be appreciated

Answer: You can use filter function to filter the array of structs then get value: from pyspark.sql import functions as F df2 = df.withColumn( "B", F.expr("filter(customer.attributes, x -> x.key = 'B')")[0]["value"] )
Question: I could not find any method that returns outer html from python playwright  page.locator(selector, **kwargs). Am I missing something? locator.inner_html(**kwargs) do exists. However, I am trying to use pandas.read_html and it fails on table locator inner html as it trips table tag. What I'm currently doing is using bs4 to parse page.content(). something like: soup = BeautifulSoup(page.content(), 'lxml') df = pd.read_html(str(soup.select('table.selector'))) 

Answer: There is no outer_html out of the box. But it's not hard to implement it: locator.evaluate("el => el.outerHTML")
Question: I am working on a project where I am using psycopg2 connection to fetch the data from the database like this, cursor = connection.execute("select * from table") cursor.fetchall()  Now after getting the data from the table, I am running some extra operations to convert the data from cursor to pandas dataframe. I am looking for some library or some more robust way to convert the data to pandas dataframe from psycopg2 connection. Any help of guidance will be appreciated. Thanks

Answer: You can use pandas sqlio module to run and save query within pandas dataframe. Let's say you have a connection of psycopg2 connection then you can use pandas sqlio like this. import pandas.io.sql as sqlio data = sqlio.read_sql_query("SELECT * FROM table", connection) # Now data is a pandas dataframe having the results of above query. data.head()  For me, sqlio pandas module is working fine. Please have a look at it and let me know if this is what you are looking for.
Question: Consider the following: from __future__ import annotations class A: def __init__(self): print("A") self.hello = "hello" # how do I type this so that the return type is A for A.bobo() # and B for B.bobo()? @classmethod def bobo(cls) -> UnknownType: return cls() class B(A): def __init__(self): print("B") super().__init__() self.world = "world" instance_of_B = B.bobo() # prints "B", then "A", and returns an instance of B  I want to type-hint the bobo class method, so that mypy can know that in the case of B's bobo method, it's not just an instance of A that's returned, but actually an instance of B. I'm really unclear on how to do that, or if it's even possible. I thought that something like Type[cls] might do the trick, but I'm not sure if that even makes syntactic sense to mypy.

Answer: You will have to use a TypeVar, thankfully, in Python 3.11 the typing.Self type is coming out. This PEP describes it in detail. It also specifies how to use the TypeVar until then.
Question: Is it possible to have a Python docstring calculated? I have a lot of repetitive things in my docstrings, so I'd like to either use f-strings or a %-style format expression. When I use an f-string at the place of a docstring  importing the module invokes the processing but when I check the __doc__ of such a function it is empty sphinx barfs when the docstring is an f-string  I do know how to process the docstrings after the import, but that doesn't work for object 'doc' strings which is recognized by sphinx but is not a real __doc__'s of the object.

Answer: Docstrings in Python must be regular string literals. This is pretty easy to test - the following program does not show the docstring: BAR = "Hello world!" def foo(): f"""This is {BAR}""" pass assert foo.__doc__ is None help(foo)  The Python syntax docs say that the docstring must be a "string literal", and the tail end of the f-string reference says they "cannot be used as docstrings". So unfortunately you must use the __doc__ attribute. However, you should be able to use a decorator to read the __doc__ attribute and replace it with whatever you want.
Question: I have a simple Dockerfile with Python and NodeJS. I install pytest, a local library and run tests: FROM nikolaik/python-nodejs:latest ADD . . RUN pip3 install --upgrade pip RUN pip3 install -e . RUN pip3 install pytest CMD ["pytest"]  However, pytest collection fails: ============================= test session starts ============================== platform linux -- Python 3.10.2, pytest-6.2.5, py-1.11.0, pluggy-1.0.0 rootdir: / collected 0 items / 1 error ==================================== ERRORS ==================================== ________________________ ERROR collecting test session _________________________ usr/local/lib/python3.10/site-packages/_pytest/runner.py:311: in from_call result: Optional[TResult] = func() usr/local/lib/python3.10/site-packages/_pytest/runner.py:341: in <lambda> call = CallInfo.from_call(lambda: list(collector.collect()), "collect") usr/local/lib/python3.10/site-packages/_pytest/main.py:690: in collect for direntry in visit(str(argpath), self._recurse): usr/local/lib/python3.10/site-packages/_pytest/pathlib.py:606: in visit yield from visit(entry.path, recurse) usr/local/lib/python3.10/site-packages/_pytest/pathlib.py:606: in visit yield from visit(entry.path, recurse) usr/local/lib/python3.10/site-packages/_pytest/pathlib.py:606: in visit yield from visit(entry.path, recurse) usr/local/lib/python3.10/site-packages/_pytest/pathlib.py:591: in visit for entry in os.scandir(path): E FileNotFoundError: [Errno 2] No such file or directory: '/dev/fd/11' =========================== short test summary info ============================ ERROR - FileNotFoundError: [Errno 2] No such file or directory: '/dev/fd/11' !!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!! =============================== 1 error in 0.25s =============================== 

Answer: I found a solution on pytest GitHub: https://github.com/pytest-dev/pytest/issues/8960  your working directory is / so pytest is attempting to recurse through everything in the filesystem (probably not what you want!)  Added WORKDIR /tests/ to the Dockerfile and the issue is fixed.
Question:  I am trying to receive data from a website which use websocket. This acts like this: websocket handshaking  Here is the code to catch data: async def hello(symb_id: int): async with websockets.connect("wss://ws.bitpin.ir/", extra_headers = request_header, timeout=15) as websocket: await websocket.send('{"method":"sub_to_price_info"}') recv_msg = await websocket.recv() if recv_msg == '{"message": "sub to price info"}': await websocket.send(json.dumps({"method":"sub_to_market","id":symb_id})) recv_msg = await websocket.recv() counter = 1 while(1): msg = await websocket.recv() print(counter, msg[:100], end='\n\n') counter+=1 asyncio.run(hello(1))   After receiving about 100 message, I am facing with this error:  websockets.exceptions.ConnectionClosedOK: code = 1000 (OK), no reason   I tried to set timeout and request headers but these were not helpful 

Answer: I solved this error by creating a task for sending PING. async def hello(symb_id: int): async with websockets.connect("wss://ws.bitpin.ir/", extra_headers = request_header, timeout=10, ping_interval=None) as websocket: await websocket.send('{"method":"sub_to_price_info"}') recv_msg = await websocket.recv() if recv_msg == '{"message": "sub to price info"}': await websocket.send(json.dumps({"method":"sub_to_market","id":symb_id})) recv_msg = await websocket.recv() counter = 1 task = asyncio.create_task(ping(websocket)) while True: msg = await websocket.recv() await return_func(msg) print(counter, msg[:100], end='\n\n') counter+=1 async def ping(websocket): while True: await websocket.send('{"message":"PING"}') print('------ ping') await asyncio.sleep(5)
Question: I need to return a value in async function. I tried to use synchronous form of return: import asyncio async def main(): for i in range(10): return i await asyncio.sleep(1) print(asyncio.run(main()))  output: 0 [Finished in 204ms] But it just return value of the first loop, which is not expexted. So changed the code as below: import asyncio async def main(): for i in range(10): yield i await asyncio.sleep(1) for _ in main(): print(_)  output: TypeError: 'async_generator' object is not iterable by using async generator I am facing with this error. How can I return a value for every loop of async function? Thanks

Answer: You need to use an async for which itself needs to be inside an async function: async def get_result(): async for i in main(): print(i) asyncio.run(get_result())
Question: I developed a custom dataset by using the PyTorch dataset class. The code is like that: class CustomDataset(torch.utils.data.Dataset): def __init__(self, root_path, transform=None): self.path = root_path self.mean = mean self.std = std self.transform = transform self.images = [] self.masks = [] for add in os.listdir(self.path): # Some script to load file from directory and appending address to relative array ... self.masks.sort() self.images.sort() def __len__(self): return len(self.images) def __getitem__(self, item): image_address = self.images[item] mask_address = self.masks[item] if self.transform is not None: augment = self.transform(image=np.asarray(Image.open(image_address, 'r', None)), mask=np.asarray(Image.open(mask_address, 'r', None))) image = Image.fromarray(augment['image']) mask = augment['mask'] if self.transform is None: image = np.asarray(Image.open(image_address, 'r', None)) mask = np.asarray(Image.open(mask_address, 'r', None)) # Handle Augmentation here return image, mask  Then I created an object from this class and passed it to torch.utils.data.DataLoader. Although this works well with DataLoader but with torch.utils.data.DataLoader2 I got a problem. The error is this:  dataloader = torch.utils.data.DataLoader2(dataset=dataset, batch_size=2, pin_memory=True, num_workers=4)  Exception: thread parallelism mode is not supported for old DataSets   My question is why DataLoader2 module was added to PyTorch what is different with DataLoader and what are its benefits? PyTorch Version: 1.10.1

Answer: You should definitely not use it DataLoader2. torch.utils.data.DataLoader2 (actually torch.utils.data.dataloader_experimental.DataLoader2) was added as an experimental "feature" as a future replacement for DataLoader. It is defined here. Currently, it is only accessible on the master branch (unstable) and is of course not documented on the official pages.
Question: Suppose I have a list, datalist which contains several examples (which are of type torch_geometric.data.Data for my use case). Each example has an attribute num_nodes For demo purpose, such datalist can be created using the following snippet of code import torch from torch_geometric.data import Data # each example is of this type import networkx as nx # for creating random data import numpy as np # the python list containing the examples datalist = [] for num_node in [9, 11]: for _ in range(1024): edge_index = torch.from_numpy( np.array(nx.fast_gnp_random_graph(num_node, 0.5).edges()) ).t().contiguous() datalist.append( Data( x=torch.rand(num_node, 5), edge_index=edge_index, edge_attr=torch.rand(edge_index.size(1)) ) )  From the above datalist object, I can create a torch_geometric.loader.DataLoader (which subclasses torch.utils.data.DataLoader) naively (without any constraints) by using the DataLoader constructor as: from torch_geometric.loader import DataLoader dataloader = DataLoader( datalist, batch_size=128, shuffle=True )  My question is, how can I use the DataLoader class to ensure that each example in a given batch has the same value for num_nodes attribute? PS: I tried to solve it and came up with a hacky solution by combining multiple DataLoader objects using the combine_iterators function snippet from here as follows: def get_combined_iterator(*iterables): nexts = [iter(iterable).__next__ for iterable in iterables] while nexts: next = random.choice(nexts) try: yield next() except StopIteration: nexts.remove(next) datalists = defaultdict(list) for data in datalist: datalists[data.num_nodes].append(data) dataloaders = ( DataLoader(data, batch_size=128, shuffle=True) for data in datalists.values() ) batches = get_combined_iterator(*dataloaders)  But, I think that there must be some elegant/better method of doing it, hence this question.

Answer: If your underlying dataset is map-style, you can use define a torch.utils.data.Sampler which returns the indices of the examples you want to batch together. An instance of this will be passed as a batch_sampler kwarg to your DataLoader and you can remove the batch_size kwarg as the sampler will form batches for you depending on how you implement it.
Question: I'm trying to use SPM to determine if a certain type is an int or an str. The following code: from typing import Type def main(type_to_match: Type): match type_to_match: case str(): print("This is a String") case int(): print("This is an Int") case _: print("\nhttps://en.meming.world/images/en/0/03/I%27ve_Never_Met_This_Man_In_My_Life.jpg") if __name__ == "__main__": test_type = str main(test_type)  returns https://en.meming.world/images/en/0/03/I%27ve_Never_Met_This_Man_In_My_Life.jpg Most of the documentation I found talks about how to test if a certain variable is an instance of a type. But not how to test if a type is of a certain type. Any ideas on how to make it work?

Answer: If you just pass a type directly, it will consider it to be a "name capture" rather than a "value capture." You can coerce it to use a value capture by importing the builtins module, and using a dotted notation to check for the type. import builtins from typing import Type def main(type_: Type): match (type_): case builtins.str: # it works with the dotted notation print(f"{type_} is a String") case builtins.int: print(f"{type_} is an Int") case _: print("\nhttps://en.meming.world/images/en/0/03/I%27ve_Never_Met_This_Man_In_My_Life.jpg") if __name__ == "__main__": main(type("hello")) # <class 'str'> is a String main(str) # <class 'str'> is a String main(type(42)) # <class 'int'> is an Int main(int) # <class 'int'> is an Int