eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I'm developing an application in Python which uses Azure Cosmos DB as the main database. At some point in the app, I need to insert bulk data (a batch of items) into Cosmos DB. So far, I've been using Azure Cosmos DB Python SDK for SQL API for communicating with Cosmos DB; however, it doesn't provide a method for bulk data insertion. As I understood, these are the insertion methods provided in this SDK, both of which only support single item insert, which can be very slow when using it in a for loop:  .upsert_item() .create_item()  Is there another way to use this SDK to insert bulk data instead of using the methods above in a for loop? If not, is there an Azure REST API that can handle bulk data insertion?

Answer: The Cosmos DB service does not provide this via its REST API. Bulk mode is implemented at the SDK layer and unfortunately, the Python SDK does not yet support bulk mode. It does however support asynchronous IO. Here's an example that may help you. from azure.cosmos.aio import CosmosClient import os URL = os.environ['ACCOUNT_URI'] KEY = os.environ['ACCOUNT_KEY'] DATABASE_NAME = 'myDatabase' CONTAINER_NAME = 'myContainer' async def create_products(): async with CosmosClient(URL, credential=KEY) as client: database = client.get_database_client(DATABASE_NAME) container = database.get_container_client(CONTAINER_NAME) for i in range(10): await container.upsert_item({ 'id': 'item{0}'.format(i), 'productName': 'Widget', 'productModel': 'Model {0}'.format(i) } )  Update: I remembered another way you can do bulk inserts in Cosmos DB for Python SDK and that is using Stored Procedures. There are examples of how to write these, including samples that demonstrate passing an array, which is what you want to do. I would also take a look at bounded execution as you will want to implement this as well. You can learn how to write them here, How to write stored procedures. Then how to register and call them here, How to use Stored Procedures. Note: these can only be used when passing a partition key value so you can only do batches within logical partitions.
Question: I see dataframe error while trying to print it within single df[ _ , _ ] form. Below are the code lines #Data Frames code import numpy as np import pandas as pd randArr = np.random.randint(0,100,20).reshape(5,4) df =pd.DataFrame(randArr,np.arange(101,106,1),['PDS','Algo','SE','INS']) print(df['PDS','SE'])  errors: Traceback (most recent call last): File "C:\Users\subro\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\pandas\core\indexes\base.py", line 3621, in get_loc return self._engine.get_loc(casted_key) File "pandas\_libs\index.pyx", line 136, in pandas._libs.index.IndexEngine.get_loc File "pandas\_libs\index.pyx", line 163, in pandas._libs.index.IndexEngine.get_loc File "pandas\_libs\hashtable_class_helper.pxi", line 5198, in pandas._libs.hashtable.PyObjectHashTable.get_item File "pandas\_libs\hashtable_class_helper.pxi", line 5206, in pandas._libs.hashtable.PyObjectHashTable.get_item KeyError: ('PDS', 'SE') The above exception was the direct cause of the following exception: Traceback (most recent call last): File "D:\Education\4th year\1st sem\Machine Learning Lab\1st Lab\python\pandas\pdDataFrame.py", line 11, in <module> print(df['PDS','SE']) File "C:\Users\subro\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\pandas\core\frame.py", line 3505, in __getitem__ indexer = self.columns.get_loc(key) File "C:\Users\subro\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\pandas\core\indexes\base.py", line 3623, in get_loc raise KeyError(key) from err KeyError: ('PDS', 'SE')

Answer: Do you mean to do this? Need to indicate the column names when creating the dataframe, and also need double square brackets df[[ ]] when extracting a slice of the dataframe import numpy as np import pandas as pd randArr = np.random.randint(0,100,20).reshape(5,4) df = pd.DataFrame(randArr, columns=['PDS', 'SE', 'ABC', 'CDE']) print(df) print(df[['PDS','SE']])  Output:  PDS SE ABC CDE 0 56 77 82 42 1 17 12 84 46 2 34 9 19 12 3 19 88 34 19 4 51 54 9 94 PDS SE 0 56 77 1 17 12 2 34 9 3 19 88 4 51 54
Question: I'm trying to connect my database to a python project using the MySQL connector. However, when using the code below, import mysql.connector mydb = mysql.connector.MySQLConnection( host="localhost", user="veensew", password="%T5687j5IiYe" ) print(mydb)  I encounter the following error：  mysql.connector.errors.ProgrammingError: Character set 'utf8' unsupported  I tried to find out why this is happening and I'm getting the same error always. MySQL Connector version - 8.0.30 I'd appreciate any help. Thank you in advanced!

Answer: I ran into the same issue. There were apparently some changes in version 8.0.30 to the way utf8_ collations are handled (see MySQL Connector release notes). I installed version 8.0.29 which fixed the issue for me. pip3 install mysql-connector-python==8.0.29
Question: I am using the new multipage feature and would like to style my multipage app and put a logo with a title on top of/before the page navigation. Here's a small example tested on Python 3.9 with streamlit==1.11.1 in the following directory structure: /Home.py /pages/Page_1.py /pages/Page_2.py  Home.py: import streamlit as st st.sidebar.markdown( "My Logo (sidebar) should be on top of the Navigation within the sidebar" ) st.markdown("# Home")  Page_1.py: import streamlit as st st.markdown("Page 1")  Page_2.py: import streamlit as st st.markdown("Page 2")  which I can run using: $ streamlit run Home.py  But this leads to the Text printed below and not above the navigation:  Is there any way to do this? Any hints are welcome! Best wishes, Cord

Answer: One option is to do it via CSS, with a function like this: def add_logo(): st.markdown( """ <style> [data-testid="stSidebarNav"] { background-image: url(http://placekitten.com/200/200); background-repeat: no-repeat; padding-top: 120px; background-position: 20px 20px; } [data-testid="stSidebarNav"]::before { content: "My Company Name"; margin-left: 20px; margin-top: 20px; font-size: 30px; position: relative; top: 100px; } </style> """, unsafe_allow_html=True, )  And then just call that function at the top of each page. That produces an effect like this:
Question: So I found this code online that would make a Gantt chart: import plotly.express as px import pandas as pd df = pd.DataFrame([ dict(Task="Job A", Start='2009-01-01', Finish='2009-02-28', Resource="Alex"), dict(Task="Job B", Start='2009-03-05', Finish='2009-04-15', Resource="Alex"), dict(Task="Job C", Start='2009-02-20', Finish='2009-05-30', Resource="Max") ]) fig = px.timeline(df, x_start="Start", x_end="Finish", y="Task", color="Resource") fig.update_yaxes(autorange="reversed") fig.show()  However, I wish to not use date and times but rather use timesteps, starting from 0 nanoseconds and so forth, what I would want is something that would look like this: import plotly.express as px import pandas as pd df = pd.DataFrame([ dict(Task="Job A", Start=0, Finish=10, Resource="Alex"), dict(Task="Job B", Start=12, Finish=24, Resource="Alex"), dict(Task="Job C", Start=5, Finish=20, Resource="Max") ]) fig = px.timeline(df, x_start="Start", x_end="Finish", y="Task", color="Resource") fig.update_yaxes(autorange="reversed") fig.show()  Is this something that would be possible using plotly or does it only use Date and time? I want to use plotly because it gives a very clean graph. I've looked in the plotly express documentation but found nothing to convert and use timesteps.

Answer: You can do this using plotly.figure_factory gantt chart and forcing the x-axis to show numbers. There are few examples here, if you need to know more about this. The code output is as shown below. import pandas as pd import plotly.figure_factory as ff df = pd.DataFrame([ dict(Task="Job A", Start=0, Finish=10, Resource="Alex"), dict(Task="Job B", Start=12, Finish=24, Resource="Alex"), dict(Task="Job C", Start=5, Finish=20, Resource="Max") ]) fig = ff.create_gantt(df, index_col = 'Resource', bar_width = 0.4, show_colorbar=True) fig.update_layout(xaxis_type='linear', autosize=False, width=800, height=400) fig.show()  Plot
Question: In my flutter app I am using the flask server for testing purpose. I started my server and run the API url in my flutter app. But SocketException: Connection refused (OS Error: Connection refused, errno = 111), address = 127.0.0.1, port = 44164. error is showing. var headers = {'Content-Type': 'application/json'}; var request = http.Request('POST', Uri.parse('http://127.0.0.1:5000/addrec')); request.body = json.encode({ "name": UploadedName, "grade": Uploadedgrade, "loaction": Uploadedlocation, "like": Uploadedlike, "admission": Uploadedadmission, "comments": Uploadedcomments, "entery_time": UploadeddataentryTime }); request.headers.addAll(headers); http.StreamedResponse response = await request.send(); if (response.statusCode == 200) { print(await response.stream.bytesToString()); } else { print(response.reasonPhrase); }  I am using actual android device for app running.

Answer: This happens because the localhost (or 127.0.0.1) on the device is only accessible to the device itself. Solution 1 You can reverse-proxy a localhost port to the Android device/emulator running adb reverse on the command prompt like so: adb reverse tcp:5000 tcp:5000  Solution 2 Use the machine's IP address where the API is running. Also, the API should be listening to the IP 0.0.0.0 to be accessible outside the localhost. Supposing the API machine's IP is 192.168.1.123 it's going to be something like: Uri.parse('http://192.168.1.123:5000/addrec')  Just take care because changing the API to listen to 0.0.0.0 is a security risk as the API is going to be accessible to the outside world.
Question: I cannot wrap my head around type hinting a Coroutine. As far as I understand, when we declare a function like so: async def some_function(arg1: int, arg2: str) -> list: ...  we effectively declare a function, which returns a coroutine, which, when awaited, returns a list. So, the way to type hint it would be: f: Callable[[int, str], Coroutine[???]] = some_function  But Coroutine generic type has 3 arguments! We can see it if we go to the typing.py file: ... Coroutine = _alias(collections.abc.Coroutine, 3) ...  There is also Awaitable type, which logically should be a parent of Coroutine with only one generic parameter (the return type, I suppose): ... Awaitable = _alias(collections.abc.Awaitable, 1) ...  So maybe it would be more or less correct to type hint the function this way: f: Callable[[int, str], Awaitable[list]] = some_function  Or is it? So, basically, the questions are:  Can one use Awaitable instead of Coroutine in the case of type hinting an async def function? What are the correct parameters for the Coroutine generic type and what are its use-cases? 

Answer: As the docs state:  Coroutine objects and instances of the Coroutine ABC are all instances of the Awaitable ABC.  And for the Coroutine type:  A generic version of collections.abc.Coroutine. The variance and order of type variables correspond to those of Generator.  Generator in turn has the signature Generator[YieldType, SendType, ReturnType]. So if you want to preserve that type information, use Coroutine, otherwise Awaitable should suffice.
Question: Can someone explain to me why is numpy round acting strange with this exact number rounding: df = pd.DataFrame({'c': [121921117.714999988675115, 445, 22]}) df = np.round(df['c'], 8) Result: 121921117.71499997 445.0 22.0 Expected: 121921117.71499999 445.0 22.0  It's obvious that the first number is not rounded well, any ideas? EDIT: Since I'm focused here on the precision, not so much on the performance, I've used python round function to solve this problem: df.applymap(round, ndigits=8) 

Answer: Check the small print 2 in the documentation of round aka around. The short answer is that round "uses a fast but sometimes inexact algorithm" and to use format_float_positional if you want to see the correct result.  >>> import pandas as pd >>> df = pd.DataFrame({'c': [121921117.714999988675115, 445, 22]}) >>> df["c"][0] 121921117.71499999 >>> round(df["c"][0],8) 121921117.71499997 >>> np.format_float_positional(df["c"][0],8) '121921117.71499999'
Question: Assignment:  Print a string which has the reverse order 'Python love We. Science Data love We'  I tried this: strg = We love Data Science. We love Python words = strg.split(" ") words.reverse() new_strg = " ".join(words) print(new_strg) >>> Python love We Science. Data love We  But the answer isn't as expected because the . after Science is not at the proper place. How to get the expected result?

Answer: Is this the output you need? Python love We. Science Data love We  Then the code is strg = 'We love Data Science. We love Python' pos = len(strg) - strg.index('.') - 2 words = [e.strip('.') for e in strg.split()] words.reverse() new_strg = ' '.join(words) print(new_strg[:pos] + '.' + new_strg[pos:])  Or another way to do it: strg = 'We love Data Science. We love Python' new_strg = [s.split()[::-1] for s in strg.split('.')][::-1] print(' '.join(new_strg[0]) + '. ' + ' '.join(new_strg[1])) #or print('{}. {}'.format(' '.join(new_strg[0]), ' '.join(new_strg[1])))  Or to raise the bar: strg = 'We love Data Science. We love Python' print('. '.join([' '.join(new_strg.split()[::-1]) for new_strg in strg.split('.')[::-1]]))
Question: I need to find a specific element in my webpage. The element may be in the page or not. This code is giving me error if the element is not visible: error_text = self.page.wait_for_selector( self.ERROR_MESSAGE, timeout=7000).inner_text()  How can I look for the element using timeout, and get a bool telling me if the element is found or not?

Answer: You have to use the page.is_visible(selector, **kwargs) for this as this returns a boolean value. Playwright Docs, bool = page.is_visible(selector, timeout=7000) print(bool) #OR if page.is_visible(selector, timeout=7000): print("Element Found") else: print("Element not Found")  You can also use expect assertion if you directly want to assert the presence of the element and fail the tests if the condition is not met. expect(page.locator("selector")).to_have_count(1, timeout=7000)
Question: I have a file on my computer that I want to upload to Google Colab. I know there are numerous ways to do this, including a from google.colab import files uploaded = files.upload()  or just uploading manually from the file system. But I want to upload that specific file without needing to choose that file myself. Something like: from google.colab import files file_path = 'path/to/the/file' files.upload(file_path)  Is there any way to do this?

Answer: Providing a file path directly rather than clicking through the GUI for an upload requires access to your local machine's file system. However, when your run cell IPython magic commands such as %pwd in Google collab, you'll notice that the current working directory shown is that of the notebook environment - not that of your machine. The way to eschew the issue are as follows. 1. Local Runtime Only local runtimes via Jupyter seems to enable such access to the local file system. This necessitates the installation of jupyterlab, a Jupyter server extension for using a WebSocket, and launching a local server. See this tutorial. 2. Google Drive In case Google Drive is convenient, you can upload files into Google Drive from your local machine without clicking through a GUI. 3. Embracing the GUI If these options seem overkill, you, unfortunately, have to stick with from google.colab import files uploaded = files.upload()  as you alluded to.
Question: Problem Setup & Goal I have a Multindexed Pandas DataFrame that looks like this: import pandas as pd df = pd.DataFrame({ 'Values':[1, 3, 4, 8, 5, 2, 9, 0, 2], 'A':['A1', 'A1', 'A1', 'A1', 'A2', 'A2', 'A3', 'A3', 'A3'], 'B':['foo', 'bar', 'fab', 'baz', 'foo', 'baz', 'qux', 'baz', 'bar'] }) df.set_index(['A','B'], inplace=True) print(df.to_string()) Values A B A1 foo 1 bar 3 fab 4 baz 8 A2 foo 5 baz 2 A3 qux 9 baz 0 bar 2  My ultimate goal is to replace all the "bar" and "baz" rows in the B column with a summed row called "other" (see below) in the simplest, most canonical Pandas way.  Values A B A1 foo 1 fab 4 other 11 A2 foo 5 other 2 A3 qux 9 other 2  Current Work I managed to figure out how to create a mask for a MultiIndex DataFrame from a similar problem to highlight the rows we want to eventually aggregate, which are in an agg_list. agg_list = ['bar', 'baz'] # Create a mask that highlights the rows in B that are in agg_list filterFunc = lambda x: x.index.get_level_values('B') in agg_list mask = df.groupby(level=['A','B']).apply(filterFunc)  This produces the expected mask: print(mask.to_string()) A B A1 bar True baz True fab False foo False A2 baz True foo False A3 bar True baz True qux False  And I know how to remove the rows I no longer need: # Remove rows in B col that are in agg_list using mask df_masked = df[[~mask.loc[i1, i2] for i1,i2 in df.index]] print(df_masked.to_string()) Values A B A1 foo 1 fab 4 A2 foo 5 A3 qux 9  But I don't know how to do the actual aggregation/sum on these rows and append it to each Multindexed row. Similar Problems/Solutions Similar problems I've seen didn't involve a Multindex DataFrame, so I can't quite use some of the solutions like this one, which has the same general idea of creating a mask and then append a summed row: threshold = 6 m = df['value'] < threshold df1 = df[~m].copy() df1.loc['Z'] = df.loc[m, 'value'].sum()  or m = df['value'] < threshold df1 = df[~m].append(df.loc[m, ['value']].sum().rename('Z')) 

Answer: Here is one way which resets the index for just B, performs a replace and aggregates the values. agg_list = ['bar', 'baz'] (df.reset_index(level=1) .replace({'B':{'|'.join(agg_list):'other'}},regex=True) .groupby(['A','B']).sum())  Another way is to create a new MultiIndex with bar and baz being replaced with other. (df.set_axis(pd.MultiIndex.from_arrays([df.index.get_level_values(0), df.index.get_level_values(1).str.replace('|'.join(agg_list),'other')])) .groupby(level=[0,1]).sum())  Output:  Values A B A1 fab 4 foo 1 other 11 A2 foo 5 other 2 A3 other 2 qux 9
Question: I'm using drf-spectacular and here's code in settings.py SPECTACULAR_SETTINGS = { 'TITLE': 'TITLE', 'VERSION': '1.0.0', 'SCHEMA_PATH_PREFIX_TRIM': True, 'PREPROCESSING_HOOKS': ["custom.url_remover.preprocessing_filter_spec"], }  in serializers.py class ChangePasswordSerilaizer(serializers.Serializer): current_password = serializers.CharField(write_only=True, min_length=8, required=True) new_password = serializers.CharField(write_only=True, min_length=8, required=True) confirm_new_password = serializers.CharField(write_only=True, min_length=8, required=True)  but still fields in request body are showing not required 

Answer: change your SPECTACULAR_SETTINGS  SPECTACULAR_SETTINGS = { 'TITLE': 'APP NAME', 'VERSION': '1.0.0', 'SCHEMA_PATH_PREFIX_TRIM': True, 'PREPROCESSING_HOOKS': ["custom.url_remover.preprocessing_filter_spec"], 'COMPONENT_SPLIT_PATCH': False, }  by default COMPONENT_SPLIT_PATCH is true in SPECTACULAR_SETTINGS so you can simply override('COMPONENT_SPLIT_PATCH': False) it to fix this problem
Question: I am currently creating a new column in a polars data frame using predictions = [10, 20, 30, 40, 50] df['predictions'] = predictions  where predictions is a numpy array or list containing values I computed with another tool. However, polars throws a warning, that this option will be deprecated. How can the same result be achieved using .with_columns()?

Answer: You can now also pass numpy arrays in directly. E.g, df = pl.DataFrame({"x": [0, 1, 2, 3, 4]}) p1 = [10, 20, 30, 40, 50] p2 = np.array(p1) df.with_columns( p1=pl.Series(p1), # For python lists, construct a Series p2=p2, # For numpy arrays, you can pass them directly ) # shape: (5, 3) # ┌─────┬─────┬─────┐ # │ x ┆ p1 ┆ p2 │ # │ --- ┆ --- ┆ --- │ # │ i64 ┆ i64 ┆ i64 │ # ╞═════╪═════╪═════╡ # │ 0 ┆ 10 ┆ 10 │ # │ 1 ┆ 20 ┆ 20 │ # │ 2 ┆ 30 ┆ 30 │ # │ 3 ┆ 40 ┆ 40 │ # │ 4 ┆ 50 ┆ 50 │ # └─────┴─────┴─────┘
Question: I have copy past code from tensorflow website's introduction to autoencoder first examplefollowing code works with mnist fashion dataset but not mine.This gives me a very long warning.Please tell me what is worng with my dataset the warning screen short of same error here x_train is my dataset: tf.shape(x_train) output <tf.Tensor: shape=(3,), dtype=int32, numpy=array([169,** **28, 28])>  here x_train is the mnist dataset: tf.shape(x_train) output<tf.Tensor: shape=(3,), dtype=int32, numpy=array([60000, 28, 28])>  My whole code to make dataset: dir_path='auto/ttt/' data=[] x_train=[] for i in os.listdir(dir_path): img=image.load_img(dir_path+'//'+i,color_mode='grayscale',target_size=(28,28)) data=np.array(img) data=data/255.0 x_train.append(data)  this is the warning: WARNING:tensorflow:Layers in a Sequential model should only have a single input tensor. Received: inputs=(<tf.Tensor 'IteratorGetNext:0' shape=(None, 28) dtype=float32>, <tf.Tensor 'IteratorGetNext:1' shape=(None, 28) dtype=float32>, <tf.Tensor 'IteratorGetNext:2' shape=(None, 28) dtype=float32>, <tf.Tensor 'IteratorGetNext:3' shape=(None, 28) dtype=float32>, <tf.Tensor 'IteratorGetNext:4' shape=(None, 28) dtype=float32>, <tf.Tensor 'IteratorGetNext:5' shape=(None, 28) dtype=float32>, <tf.Tensor 'IteratorGetNext:6' shape=(None, 28) dtype=float32>, <tf.Tensor 'IteratorGetNext:7' shape=(None, 28) dtype=flo...  also this value error (same warning): ValueError: Exception encountered when calling layer "sequential_4" (type Sequential). Layer "flatten_2" expects 1 input(s), but it received 169 input tensors. Inputs received: [<tf.Tensor 'IteratorGetNext:0' shape=(None, 28) dtype=float32>, <tf.Tensor 'IteratorGetNext:1' shape=(None, 28) dtype=float32>, <tf.Tensor 'IteratorGetNext:2' shape=(None, 28) dtype=float32>, <tf.Tensor 'IteratorGetNext:3' shape=(None, 28) dtype=float32>, <tf.Tensor 'IteratorGetNext:4' shape=(None, 28) dtype=float32>, <tf.Tensor 'IteratorGetNext:5' shape=(None, 28) dtype=float32>, <tf.Tensor 'IteratorGetNext:6' shape=(None, 28) dtype=float32>, <tf.Tensor 'IteratorGetNext:7' shape=(None, 28) dtype=float32>, <tf.Tensor 'IteratorGetNext:8' shape=(None, 28) dtype=float3... 

Answer: The model.fit() is given a list of arrays as input. A list of arrays is generally passed to fit() when a model has multiple inputs. In this case, the fit() method is treating each array as an input, resulting in the error. Please convert the data to a tensor as follows and try again. x_train=tf.convert_to_tensor(x_train)  Refer to the gist for complete code.
Question: I am trying to create a timeseries in Redis using python like so:  import redis connection_redis = redis.Redis(host='127.0.0.1', port=6379) connection_redis.ts().create('ts', retention_msecs=0)  but I get the following error: ResponseError: unknown command 'TS.CREATE'. I have been searching for a way to solve this problem but I haven't found anything. I am running redis in a docker. Thank you :)!

Answer: The Redis docker image does not contain any Redis module. You can use the Redis Stack docker image. redis/redis-stack-server contains the RediSearch, RedisJSON, RedisGraph, RedisTimeSeries, and RedisBloom modules. redis/redis-stack also contains RedisInsight. Update, October 2024 From a Redis Blog Post:  Redis 8 introduces seven new data structures —JSON, time series, and five probabilistic types— along with the fastest and most scalable Redis query engine to date. These capabilities, previously only available separately through Redis Stack or our Software and Cloud offerings, are now built natively into Redis Community Edition.  You can now simply use Redis docker image (version 8.0-M01 or later).