eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: When I open a server with gunicorn for my flask app, it automatically opens the development server of flask, without giving me any errors: $ gunicorn3 --workers=1 main:app \[2023-10-16 19:46:13 +0000\] \[1061\] \[INFO\] Starting gunicorn 20.1.0 \[2023-10-16 19:46:13 +0000\] \[1061\] \[INFO\] Listening at: http://127.0.0.1:8000 (1061) \[2023-10-16 19:46:13 +0000\] \[1061\] \[INFO\] Using worker: sync \[2023-10-16 19:46:13 +0000\] \[1062\] \[INFO\] Booting worker with pid: 1062 * Serving Flask app 'app' * Debug mode: off 2023-10-16 19:46:14,209:INFO - WARNING: This is a development server. Do not use it in a production deployment. Use a production WSGI server instead. * Running on http://127.0.0.1:5000 2023-10-16 19:46:14,209:INFO - Press CTRL+C to quit  I already have other servers with gunicorn and I don't understand this behavior, does someone know? I've changed the port and the host, but the outcome is the same, also with the debug mode and the number of workers of gunicorn3.

Answer: If you have the following lines in your main application script then it will cause Flask's development server to start if __name__ == '__main__': app.run()   You could comment out the above part Make sure that you don't have any environment variables like FLASK_ENV=development or FLASK_APP=app.py Confirm that main:app correctly points to your Flask application object
Question: I created a dictionary, named dict_dif and loaded nested data into the same. The nested structure of this dictionary has the following structure:  Variables (four different variables) Regions (eleven different regions for each of the four variables) Data points (60 data points per region).  The dictionary looks as follows (where I shorten the variables, regions and data points to provide a small and better overview): 'variable_one': {'V1': [0.2883961843782473, -0.2564336831277686, -0.1502803806437477, -0.3404171090176428, 'V2': [0.3053319292730037, -0.12252840653652636, -0.18759593722968465, -0.04368054841152297, 'variable_two': {'V1': [0.05129822246425414, ...  Aim: I would like to sort the numeric values from the lowest to the highest for each variable and region, respectively. For example, in the sorted dictionary the numeric values should span from the lowest to the highest for region V1, for region V2, and so on individually for each variable. I tried the following code (among others) that I found on the internet: sorted_dict = {var : dict(sorted(roi.items(), key = lambda ele: ele[1])) for var, roi in dict_dif.items()}  However, that code seems to sort the regions based on each region’s first value. Is there a way to adjust this code above (or a different code) so that the order of the variables and regions remains the same, while only sorting the regions’ values?

Answer: If I understand correctly, you want to sort the numeric values for each variable sorted_dict = { variable: {region: sorted(values) for region, values in regions.items()} for variable, regions in dict_dif.items() }
Question: I am using this github codes (https://github.com/nageshsinghc4/-Dynamic-Time-Warping-DTW-/blob/main/Dynamic_Time_Warping(DTW).ipynb) to calculate Dynamic Time Warping. However, when run dtw_distance, warp_path = fastdtw(x, y, dist=euclidean), I get an error that "ValueError: Input vector should be 1-D.". I have same problem with this github codes (https://github.com/ElsevierSoftwareX/SOFTX-D-22-00246) too. Here is the codes, import pandas as pd import numpy as np # Plotting Packages import matplotlib.pyplot as plt import seaborn as sbn import matplotlib as mpl mpl.rcParams['figure.dpi'] = 150 savefig_options = dict(format="png", dpi=150, bbox_inches="tight") # Computation packages from scipy.spatial.distance import euclidean from fastdtw import fastdtw def compute_euclidean_distance_matrix(x, y) -> np.array: """Calculate distance matrix This method calcualtes the pairwise Euclidean distance between two sequences. The sequences can have different lengths. """ dist = np.zeros((len(y), len(x))) for i in range(len(y)): for j in range(len(x)): dist[i,j] = (x[j]-y[i])**2 return dist def compute_accumulated_cost_matrix(x, y) -> np.array: """Compute accumulated cost matrix for warp path using Euclidean distance """ distances = compute_euclidean_distance_matrix(x, y) # Initialization cost = np.zeros((len(y), len(x))) cost[0,0] = distances[0,0] for i in range(1, len(y)): cost[i, 0] = distances[i, 0] + cost[i-1, 0] for j in range(1, len(x)): cost[0, j] = distances[0, j] + cost[0, j-1] # Accumulated warp path cost for i in range(1, len(y)): for j in range(1, len(x)): cost[i, j] = min( cost[i-1, j], # insertion cost[i, j-1], # deletion cost[i-1, j-1] # match ) + distances[i, j] return cost # Create two sequences x = [7, 1, 2, 5, 9] y = [1, 8, 0, 4, 4, 2, 0] dtw_distance, warp_path = fastdtw(x, y, dist=euclidean) cost_matrix = compute_accumulated_cost_matrix(x, y)  When I run the code, I get the following error that, ValueError: Input vector should be 1-D.

Answer: There's a conflict here between what SciPy is expecting and what FastDTW is expecting. FastDTW is expecting to compare one element at a time. SciPy is expecting to get an entire vector at once. Here's what FastDTW says about the dist argument. (Source.)  dist : function or int The method for calculating the distance between x[i] and y[j]. If dist is an int of value p > 0, then the p-norm will be used. If dist is a function then dist(x[i], y[j]) will be used. If dist is None then abs(x[i] - y[j]) will be used.  Here's what SciPy says about the euclidean function:  Computes the Euclidean distance between two 1-D arrays.  The FastDTW documentation I just quoted gives you a way to solve this. A p-norm with p=2 is equivalent to the Euclidean norm. (Source.) Therefore, the simplest way to solve this problem is to provide dist=2. dtw_distance, warp_path = fastdtw.fastdtw(x, y, dist=2)
Question: I am writing some csv data to a csv gzip file in python as follows import csv import gzip import io csv_rows=[["cara","vera","tara"],["rar","mar","bar"],["jump","lump","dump"]] mem_file = io.BytesIO() with gzip.GzipFile(fileobj=mem_file,mode="wb") as gz: with io.TextIOWrapper(gz, encoding='utf-8') as wrapper: writer = csv.writer(wrapper) writer.writerows(csv_rows) gz.write(mem_file.getvalue()) gz.close() mem_file.seek(0)  When I gunzip the file the first column in the first row is a strange set of characters and causes the first row to actually have 4 columns The 2nd and 3rd rows are ok I have tried different data and always see this behavior in the first column of the first row. What is wrong with the code? For reference here it is what I see in the gunzipped file ?‹? #,eÿcara,vera,tara rar,mar,bar jump,lump,dump 

Answer: Remove this line: gz.write(mem_file.getvalue())  (also it isgetvalue(), not getValue()). The csv.writer() already uses io.TextIOWrapper to write to gzip.GzipFile, so there's no need to write to it again.
Question: It seems the problem is not difficult, but somehow I am not able to make it work. My problem is as follows. I have a dataframe say as follows:  dfin A B C a 1 198q24 a 2 128q6 a 6 1456 b 7 67q22 b 1 56 c 3 451q2 d 11 1q789  So now what I want to do is as follows, whenever the script will encounter a 'q', it will split the values and create a separate column with values starting from 'q'. The part before 'q' will remain in the original ( or maybe can crate a new column). So my desired output should be as follows:  dfout A B C D a 1 198 q24 a 2 128 q6 a 6 1456 b 7 67 q22 b 1 56 c 3 451 q2 d 11 1 q789  So what I have tried till now is as follows:  dfout = dfin.replace('\q\d*', '', regex=True)  Its creating one column without q, but I am not able to create the column D and not working as expected. Any help/ideas will help and be appreciated.

Answer: import pandas as pd def get_input() -> pd.DataFrame: csv_text = """ a 1 198q24 a 2 128q6 a 6 1456 b 7 67q22 b 1 56 c 3 451q2 d 11 1q789 """.strip() return pd.DataFrame(map(str.split, csv_text.splitlines()), columns=["a", "b", "c"]) def split_on_q(df_in: pd.DataFrame) -> pd.DataFrame: df = df_in.c.str.split("q", expand=True) df_out = df_in.copy() df_out["c"] = df[0] df_out["d"] = _prepend_q(df[1]) return df_out def _prepend_q(series: pd.Series) -> pd.Series: return series.apply(lambda s: None if s is None else f"q{s}") if __name__ == "__main__": print(split_on_q(get_input()))  Output:  a b c d 0 a 1 198 q24 1 a 2 128 q6 2 a 6 1456 None 3 b 7 67 q22 4 b 1 56 None 5 c 3 451 q2 6 d 11 1 q789
Question: I encountered a rather unexpected result today with a pandas dataframe. My script takes genome sequence data (in fasta format) as input and calculates several basic metrics. I store those metrics in a pandas dataframe. The script starts by defining an empty dataframe with headers: stats_df = pd.DataFrame(columns=['Assembly','Size','#Contigs','#Contigs > 3000','N50','Longest_contig'])  The main body of the script then loops through all genome files and calculates all the metrics listed in the headers. That all works just fine. I then add the metrics for the genome assembly to the stats_df dataframe using these two lines: new_df_row = pd.DataFrame({'Assembly':[assembly_name],'Size':[assembly_size_in_Mbp],'#Contigs':[num_of_contigs],'#Contigs > 3000':[greater_than_3000_count],'N50':[n50],'Longest_contig':[longest_contig]}) stats_df = pd.concat([stats_df,new_df_row],ignore_index=True)  The unexpected behaviour is when I view stats_df the columns are ordered: #Contigs, #Contigs > 3000, Assembly, Longest_contig, N50, Size. This is different to the order of the entries in the empty dataframe a the start and in each new row added. The metrics are all in the right place, I'm just wondering what causes the columns to move around like that?

Answer: I think the problem is that for new_df_row you're using a dictionary to create the data frame. Dictionaries are unordered (in Python 3.6 and below), so the keys do not have any fix order. This article and also this question explain the issue nicely. To fix it, you could upgrade to Python 3.7 which has ordered dictionaries.
Question: My input is this list : my_list = [[3, 4, -1], [0, 1], [-2], [7, 5, 8]]  I need to sum the nested list i and pad it to the right of the list i+1, it's like a relay course. I have to mention that the original list should be untouched and I'm not allowed to copy it. wanted = [[3, 4, -1], [0, 1, 6], [-2, 7], [7, 5, 8, 5]]  I tried the code below but I got a list with more elements than my original one : from itertools import pairwise wanted = [] for left, right in pairwise(my_list): wanted.extend([left, right + [sum(left)]]) print(wanted) [[3, 4, -1], [0, 1, 6], [0, 1], [-2, 1], [-2], [7, 5, 8, -2]]  Can you guys explain what's happening please or suggest another solution ?

Answer: You can try itertools.accumulate: from itertools import accumulate my_list = [[3, 4, -1], [0, 1], [-2], [7, 5, 8]] print(list(accumulate(my_list, lambda l1, l2: [*l2, sum(l1)])))  Prints: [[3, 4, -1], [0, 1, 6], [-2, 7], [7, 5, 8, 5]]   EDIT: As @KellyBundy pointed in the comments, the first element in the list is shared. So if it's a problem, make a copy from it e.g.: accumulate([my_list[0][:], *my_list[1:]], lambda l1, l2: [*l2, sum(l1)])
Question: I am given a list to put into a function. The function is then suppose to return the amount of times each number was in the function. So for example, if the given list is [15,15,15,6,6,6,7,7], the output list should be [3,15,3,6,2,7]. def encode_rle(flat_data): set_list = set(flat_data) flat_data_converted = [] for value in set_list: flat_data_converted.append(value) list = [] for number in flat_data_converted: amount = flat_data.count(number) list.append(amount) new_list = flat_data_converted + list  This is what I have done so far, but the numbers in the list are not attached to their occurrence amount.

Answer: def encode_rle(flat_data): result = [] i = 0 n = len(flat_data) while i < n: count = 1 while i + 1 < n and flat_data[i] == flat_data[i + 1]: count += 1 i += 1 result.extend([count, flat_data[i]]) i += 1 return result flat_data = [15,15,15,6,6,6,7,7] print(flat_data) print(encode_rle(flat_data))  Expected output: [3, 15, 3, 6, 2, 7]
Question: So I am pretty new to asynchronous programming in python3. I am creating aws sqs client using types_aiobotocore_sqs and aiobotocore libraries. But I am getting the error mentioned in the title. Following is the code implementation url.py file: @api_view(["GET"]) def root_view(request): async def run_async(): sqs_helper = await Helper.create(QueueName.TEST_QUEUE.value) message = await sqs_helper.get_queue_url() return message response = asyncio.run(run_async()) return Response(str(response))  sqs_helper file class Helper: def __init__(self, queue_name: str) -> None: self.queue_name = queue_name self.access_key_id = env("AWS_ACCESS_KEY_ID") self.secret_access_key = env("AWS_SECRET_ACCESS_KEY") self.region_name = env("REGION_NAME") self.sqs = None @classmethod async def create(cls, queue_name): instance = cls(queue_name) await instance.setup() return instance async def setup(self): self.sqs_client = await self.create_sqs_client() self.sqs = Sqs(self.sqs_client, self.queue_name) async def create_sqs_client(self): session = get_session() async with session.create_client('sqs', region_name=self.region_name, aws_access_key_id=self.access_key_id, aws_secret_access_key=self.secret_access_key) as client: client: SQSClient print("TYPE" + str(client)) return client async def get_queue_url(self): return await self.sqs.get_queue_url()  Sqs client which initiating the queue class Sqs: def __init__(self, client: SQSClient, queue_name: str) -> None: self.queue_name = queue_name self._queue_url = "" self.client = client async def get_queue_url(self) -> str: if not self._queue_url: try: response = await self.client.get_queue_url(QueueName=self.queue_name) self._queue_url = response["QueueUrl"] except ClientError as err: if ( err.response.get("Error", {}).get("Code") == "AWS.SimpleQueueService.NonExistentQueue" ): raise QueueDoesNotExist( f"Queue {self.queue_name} does not exist" ) from err raise err return self._queue_url  I am getting the particular error at response = await self.client.get_queue_url(QueueName=self.queue_name) in Sqs class which in the third code snippet.

Answer: When you use async with session.create_client(...) as client, the client object is limited to that context and will be None outside of it. You should assign it directly instead to ensure it's accessible outside that context: async def create_sqs_client(self): session = get_session() client = await session.create_client('sqs', region_name=self.region_name, aws_access_key_id=self.access_key_id, aws_secret_access_key=self.secret_access_key) print("TYPE" + str(client)) return client  Make sure to also handle the closing of the client manually since you're not using async with which would do that for you.
Question: Suppose I create a database and add desired tables, columns with this code: import sqlite3 connection = sqlite3.connect("clients.db", check_same_thread=False) cursor = connection.cursor() cursor.execute("CREATE TABLE IF NOT EXISTS customers(user_id INT, user_name, dt_string)") cursor.execute("""INSERT INTO customers VALUES(?, ?. ?);""", (user_id, user_name, dt_string)) connection.commit()  Now I know I can delete this row upon entry with this code cursor.execute("""DELETE FROM customers WHERE user_id=?;""", (user_id,) ) connection.commit()  How can I wait for a certain amount of time before removing this entry? I know it's achievable using time.sleep and threading but I wanted to know if there is any other way.

Answer: Understand that sqlite is serverless. That is its whole purpose. So there is nothing doing anything fancy in the background. It is not like SQL servers, that may have some procedures and stuff that could implement this kind of temporality (I am not aware of anything like that in MySql neither. But, I am not well versed into it, so that doesn't prove that it doesn't exist. And at least it could exist. Redis, for example have such things). So here, if someone make the row disappear after a while, it is you. Since there is nobody else. Sqlite is just a library to open and read files, not an API to talk with "someone else", that is to a server. So, one method, for example, could be to use a timer from threading import Timer def delUser(userid): cursor = connection.cursor() cursor.execute("""DELETE FROM customers WHERE user_id=?;""", (user_id,) ) connection.commit() #... Timer(10, delUser, (user_id,)).start()  But of course, you'll have to deal with multithreading caveout, if the main thread may still be writing during that time. You've already added the check_same_thread=False flag (I don't know why btw. I needed it, but you didn't so far. Probably because you've already tried similar solutions). But you also have to handle some semaphores or anyway to ensure that there is only one thread writing at a time. Quote from documentation  When using multiple threads with the same connection writing operations should be serialized by the user to avoid data corruption  There is a discussion about this in another question
Question: I am facing an issue with my Dockerized Django application. I am using the following Dockerfile to build my application: FROM python:alpine ENV PYTHONDONTWRITEBYTECODE 1 ENV PYTHONUNBUFFERED 1 ENV DJANGO_SUPERUSER_PASSWORD datahub RUN mkdir app WORKDIR /app COPY ./app . RUN mkdir -p volumes RUN apk update RUN apk add --no-cache gcc python3-dev musl-dev mariadb-dev RUN pip3 install --upgrade pip RUN pip3 install -r requirements.txt RUN apk del gcc python3-dev musl-dev CMD python3 manage.py makemigrations --noinput &&\ while ! python3 manage.py migrate --noinput; do sleep 1; done && \ python3 manage.py collectstatic --noinput &&\ python3 manage.py createsuperuser --user datahub --email admin@localhost --noinput;\ python3 manage.py runserver 0.0.0.0:8000  In my requirements.txt file: kafka-python==2.0.2  When I run my application inside the Docker container, I encounter the following error: ModuleNotFoundError: No module named 'kafka.vendor.six.moves'  Compelete Error: Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/usr/local/lib/python3.12/site-packages/kafka/__init__.py", line 23, in <module> from kafka.consumer import KafkaConsumer File "/usr/local/lib/python3.12/site-packages/kafka/consumer/__init__.py", line 3, in <module> from kafka.consumer.group import KafkaConsumer File "/usr/local/lib/python3.12/site-packages/kafka/consumer/group.py", line 13, in <module> from kafka.consumer.fetcher import Fetcher File "/usr/local/lib/python3.12/site-packages/kafka/consumer/fetcher.py", line 19, in <module> from kafka.record import MemoryRecords File "/usr/local/lib/python3.12/site-packages/kafka/record/__init__.py", line 1, in <module> from kafka.record.memory_records import MemoryRecords, MemoryRecordsBuilder File "/usr/local/lib/python3.12/site-packages/kafka/record/memory_records.py", line 27, in <module> from kafka.record.legacy_records import LegacyRecordBatch, LegacyRecordBatchBuilder File "/usr/local/lib/python3.12/site-packages/kafka/record/legacy_records.py", line 50, in <module> from kafka.codec import ( File "/usr/local/lib/python3.12/site-packages/kafka/codec.py", line 9, in <module> from kafka.vendor.six.moves import range ModuleNotFoundError: No module named 'kafka.vendor.six.moves'  I have already tried updating the Kafka package, checking dependencies, and installing the six package manually. However, the issue still persists. Can anyone provide insights on how to resolve this error? Thank you in advance for your help!

Answer: This appears to be a Python 3.12 issue, I have the same error but in an entirely different context. Instead of FROM python:alpine I suggest you use FROM python:3.11 3.12 is still very new are there are many projects still trying to work out issues.
Question: An class instance should have access to the classes attributes. __bases__ as per my understand is a class attribute If I have a class called C1, I can call C1.__bases__ but if I define an instance of C1, obj1 = C1(), obj1.__bases__ does not work? I am surprised because using attribute notation, it should start a search in C1's namespace and find __bases__

Answer: Standard attribute lookup searches through an object's __dict__ and those of its class and its class's ancestors. A class's __bases__ attribute isn't defined in its __dict__. __bases__ is managed by a descriptor in type.__dict__, which handles the attribute lookup by retrieving the tuple from an internal field in the class's memory layout. Since __bases__ isn't in C1.__dict__, looking up __bases__ on an instance of C1 doesn't find it. You have to look it up on C1 itself.
Question: I have a situation where I have an existing file. I want to compress this file using gzip and get the base64 encoding of this file and use this string for latter operations including sending as part of data in an API call. I have the following code which works fine: import base64 import gzip base64_string_to_use_later = None with open('C:\\test.json', 'rb') as orig_file: with gzip.open('C:\\test.json.gz', 'wb') as zipped_file: zipped_file.writelines(orig_file) with gzip.open('C:\\test.json.gz', 'rb') as zipped_file: base64_string_to_use_later = base64.b64encode(zipped_file.read())  This code will take the existing file, create a compressed version and write this back to the file system. The second block takes the compressed file, opens it and fetches the base 64 encoded version. Is there a way to make this more elegant to compress the file in memory and retrieve the base64 encoded string in memory?

Answer: Use gzip.compress() to compress the data in memory instead of writing to a file. import base64 import gzip with open('C:\\test.json', 'rb') as orig_file: base64_string_to_use_later = base64.b64encode(gzip.compress(orig_file.read()))
Question: def apply_km_cluster(df): conditions = [ (df['value'].between(0, 20000)), # Between 0 and 20,000 (df['value'].between(20000, 50000)), # Between 20,000 and 50,000 (df['value'].between(50000, 100000)), # Between 50,000 and 100,000 (df['value'].between(100000, 200000)), # Between 100,000 and 200,000 (df['value'] >= 200000) # Greater than or equal to 200,000 ] values = [1, 2, 3, 4, 5] # Assign values for each condition. df['cluster'] = np.select(conditions, values) return df  Hi, i have df and value column be Int64 type and if i use np.select like this i get "TypeError: invalid entry 0 in condlist: should be boolean ndarray". After some research, i convert int64 type and error be resolved why Int64 get this error i cant understand, any explian. I tried Int64 and int64 types in np.select and expect it to work for both.

Answer: np.select has a test # If cond array is not an ndarray in boolean format or scalar bool, abort. for i, cond in enumerate(condlist): if cond.dtype.type is not np.bool_: raise TypeError( 'invalid entry {} in condlist: should be boolean ndarray'.format(i))  If df is a dataframe with Int64 dtype: In [316]: df.info() <class 'pandas.core.frame.DataFrame'> RangeIndex: 6 entries, 0 to 5 Data columns (total 1 columns): # Column Non-Null Count Dtype --- ------ -------------- ----- 0 value 6 non-null Int64 dtypes: Int64(1) memory usage: 182.0 bytes In [317]: df['value'].between(0, 20000).values Out[317]: <BooleanArray> [False, False, False, True, True, False] Length: 6, dtype: boolean In [318]: _.dtype Out[318]: BooleanDtype  The values is BooleanArray with a pandas extension dtype. At least for now, np.select cannot handle conditions that yield extended dtypes. I believe the pandas extended dtype are still considered experimental. https://pandas.pydata.org/docs/user_guide/boolean.html BooleanArray is currently experimental. Its API or implementation may change without warning.