eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: Trying to do an update_item which is supposed to create new attributes if it doesn't find existing ones (according to documentation) but I am getting a Sytax error. I have been wracking my brain all day trying to figure out why I am getting this and I can't seem to get past this. Thank you for any help Error I am getting:  ClientError: An error occurred (ValidationException) when calling the UpdateItem operation: ExpressionAttributeValues contains invalid key: Syntax error; key: "var4"  MyCode: dynamodb = boto3.resource('dynamodb') table = dynamodb.Table('contacts') table.update_item( Key={'email': emailID}, UpdateExpression=SET last_name = :var0, address_1_state = :var1, email_2 = :var2, phone = :var3, phone_2 = :var4 ExpressionAttributeValues={ 'var0': 'Metzger', 'var1': 'CA', 'var2': 'none', 'var3': '949 302-9072', 'var4': '818-222-2311' } ) 

Answer: just change the section like the following -  ExpressionAttributeValues={ ':var0': 'Metzger', ':var1': 'CA', ':var2': 'none', ':var3': '949 302-9072', ':var4': '818-222-2311' }  Hope the code will work then :)
Question: I would like to use a diverging colormap to color the background of a pandas dataframe. The aspect that makes this trickier than one would think is the centering. In the example below, a red to blue colormap is used, but the middle of the colormap isn't used for values around zero. How to create a centered background color display where zero is white, all negatives are a red hue, and all positives are a blue hue? import pandas as pd import numpy as np import seaborn as sns np.random.seed(24) df = pd.DataFrame() df = pd.concat([df, pd.DataFrame(np.random.randn(10, 4)*10, columns=list('ABCD'))], axis=1) df.iloc[0, 2] = 0.0 cm = sns.diverging_palette(5, 250, as_cmap=True) df.style.background_gradient(cmap=cm).set_precision(2)   The zero in the above display has a red hue and the closest to white background is used for a negative number.

Answer: import pandas as pd import numpy as np import seaborn as sns import matplotlib.pyplot as plt from matplotlib import colors np.random.seed(24) df = pd.DataFrame() df = pd.concat([df, pd.DataFrame(np.random.randn(10, 4)*10, columns=list('ABCD'))], axis=1) df.iloc[0, 2] = 0.0 cm = sns.diverging_palette(5, 250, as_cmap=True) def background_gradient(s, m, M, cmap='PuBu', low=0, high=0): rng = M - m norm = colors.Normalize(m - (rng * low), M + (rng * high)) normed = norm(s.values) c = [colors.rgb2hex(x) for x in plt.cm.get_cmap(cmap)(normed)] return ['background-color: %s' % color for color in c] even_range = np.max([np.abs(df.values.min()), np.abs(df.values.max())]) df.style.apply(background_gradient, cmap=cm, m=-even_range, M=even_range).set_precision(2)
Question: Given any DataFrame 2-dimensional, you can call eg. df.sample(frac=0.3) to retrieve a sample. But this sample will have completely shuffled row order. Is there a simple way to get a subsample that preserves the row order?

Answer: What we can do instead is use df.sample(), and then sort the resultant index by the original row order. Appending the sort_index() call does the trick. Here's my code: df = pd.DataFrame(np.random.randn(100, 10)) result = df.sample(frac=0.3).sort_index()  You can even get it in ascending order. Documentation here.
Question: How can I change the index values of a Pandas Series from the regular integer value that they default to, to values within a list that I have? e.g. x = pd.Series([421, 122, 275, 847, 175]) index_values = ['2014-01-01', '2014-01-02', '2014-01-03', '2014-01-04', '2014-01-05']  How do I get the dates in the index_values list to be the indexes in the Series that I've created?

Answer: You can assign index values by list: x.index = index_values print(x) 2014-01-01 421 2014-01-02 122 2014-01-03 275 2014-01-04 847 2014-01-05 175 dtype: int64
Question: I am having some issues with my django app since updating my dependencies. Here aer my installed apps: INSTALLED_APPS = [ 'graphene_django', 'rest_framework', 'corsheaders', 'django.contrib.admin', 'django.contrib.auth', 'django.contrib.contenttypes', 'django.contrib.sessions', 'django.contrib.messages', 'django.contrib.staticfiles', 'dojo_manager.dojo', ]  and my requirements.txt: aniso8601==8.0.0 asgiref==3.2.3 Django==3.0.2 django-cors-headers==3.2.0 django-filter==2.2.0 django-graphql-jwt==0.3.0 djangorestframework==3.11.0 djangorestframework-jwt==1.11.0 graphene==2.1.8 graphene-django==2.8.0 graphene-django-extras==0.4.8 graphql-core==3.0.1 graphql-relay==3.0.0 pip-upgrade-outdated==1.5 pipupgrade==1.5.2 promise==2.3 PyJWT==1.7.1 python-dateutil==2.8.1 pytz==2019.3 Rx==3.0.1 singledispatch==3.4.0.3 six==1.13.0 sqlparse==0.3.0  I am getting  ImportError: cannot import name 'ResolveInfo' from 'graphql' (E:\Ben\GitHub-Repos\dojo-manager\env\lib\site-packages\graphql\__init__.py)  I am aware of https://github.com/graphql-python/graphene-django/issues/737 and https://github.com/graphql-python/graphene/issues/546 , none of which seem to solve it in my case. Any help greatly appreciated. 

Answer: Ok I was able to fix it by downgrading graphql-core==3.0.1 to graphql-core<3 (and all the depencencies). I must have missed the errors when performing pip install -r requirements.txt
Question: I was trying to upgrade to the latest version of a package I had installed with pip, but for some reason it won't get the latest version. I've tried uninstalling the package in question, or even reinstalling pip entirely, but it still refuses to get the latest version from PyPI. When I try to pin the package version (e.g. pip install package==0.10.0) it says that it "Could not find a version that satisfies the requirement package==0.10.0 (from versions: ...)" pip search package even acknowledges that the installed version isn't the latest, labeling the two versions for me. I've seen other questions with external files or local versions, but I've tried the respective solutions (--allow-external doesn't exist anymore, and --no-cache-dir doesn't help) and I'm still stuck on the older version.

Answer: I was trying to upgrade Quart. Maybe other packages have something else going on. In this particular case, Quart had dropped support for Python 3.6 (the version I had installed) and only supported 3.7 or later. (This was a fairly recent change to the project, so I just didn't see the news.) However, when attempting to install a package only supported by a later Python, pip doesn't really explain why it couldn't find a version to satisfy the requirement - instead, it just lists all the versions that should work with the current Python, without indicating that more exist and just can't be installed. The only real options to fix are:  Update your Python to meet the package's requirements Ask/help the maintainer to backport the package to the version you have.
Question: I have 2 y points for each x points. I can draw the plot with this code: import matplotlib.pyplot as plt x = [0, 2, 4, 6] y = [(1, 5), (1, 3), (2, 4), (2, 7)] plt.plot(x, [i for (i,j) in y], 'rs', markersize = 4) plt.plot(x, [j for (i,j) in y], 'bo', markersize = 4) plt.xlim(xmin=-3, xmax=10) plt.ylim(ymin=-1, ymax=10) plt.xlabel('ID') plt.ylabel('Class') plt.show()  This is the output:  How can I draw a thin line connecting each y point pair? Desired output is: 

Answer: just add plt.plot((x,x),([i for (i,j) in y], [j for (i,j) in y]),c='black')
Question: print("Hello") def world(): print("Hello") world()  Gets corrected to: print("Hello") def world(): print("Hello") world()  I have tried to:  Reinstall Virtual Studio Code Reinstall Python 3.8 Computer Reboot Using other formatters like Black and yapf but got the same result 

Answer: Because autopep8 follows PEP8 which suggests 2 blank lines around top-level functions.  Surround top-level function and class definitions with two blank lines.
Question: I am trying to parse MongoDB data to a pydantic schema but fail to read its _id field which seem to just disappear from the schema. The issue is definitely related to the underscore in front of the object attribute. I can't change _id field name since that would imply not parsing the field at all. Please find below the code I use (using int instead of ObjectId for the sake of simplification) from pydantic import BaseModel class User_1(BaseModel): _id: int data_1 = {"_id": 1} parsed_1 = User_1(**data_1) print(parsed_1.schema()) class User_2(BaseModel): id: int data_2 = {"id": 1} parsed_2 = User_2(**data_2) print(parsed_2.schema())  User_1 is parsed successfully since its _id field is required but can't be read afterwards. User_2 works in the above example by fails if attached to Mongo which doesn't provide any id field but _id. Output of the code above reads as follows: User_1 {'title': 'User_1', 'type': 'object', 'properties': {}} User_2 {'title': 'User_2', 'type': 'object', 'properties': {'id': {'title': 'Id', 'type': 'integer'}}, 'required': ['id']} 

Answer: you need to use an alias for that field name from pydantic import BaseModel, Field class User_1(BaseModel): id: int = Field(..., alias='_id')  See the docs here.
Question: I have the following python code: import dash import dash_html_components as html import pandas as pd df = pd.read_csv('https://raw.githubusercontent.com/dougmellon/rockies_dash/master/rockies_2019.csv') def generate_table(dataframe, max_rows=10): return html.Table( # Header [html.Tr([html.Th(col) for col in dataframe.columns])] + # Body [html.Tr([ html.Td(dataframe.iloc[i][col]) for col in dataframe.columns ]) for i in range(min(len(dataframe), max_rows))] ) external_stylesheets = ['https://codepen.io/chriddyp/pen/bWLwgP.css'] app = dash.Dash(__name__, external_stylesheets=external_stylesheets) app.layout = html.Div(children=[ html.H4('Batting Stats (2019)'), generate_table(df) ]) if __name__ == '__main__': app.run_server(debug=True)  which is pulling data from this csv file (github):  When I run the following code,  python app.py  It displays data with greater than three decimals - which isn't in my csv file.  I have tried three or four times to reenter the data manually and re-upload the CSV to github but for some reason there is still data with greater than three decimals. Does anyone have any suggestions as to how I could possibly fix this issue?

Answer: I would check this for the dataframe and maybe it might help - How to display pandas DataFrame of floats using a format string for columns? or simply try- pd.options.display.float_format = '${:.2f}'.format  I just read in one of the DashTable forums that - you can format data in pandas dataframe and DataTable will display them that way as they are. For example: To display percent value: table_df['col_name']=table_df['col_name'].map('{:,.2f}%'.format)  To display float type without decimal part: table_df['col_name']=table_df['col_name'].map("{:,.0f}".format)
Question: I want to work with SharePoint sites using Python. For now, I just want to ls on the site. I immediately ran into an issue where I need site id to do anything with it. How do I get site id from the site name? I did retrieve it using powershell: $uri = "https://graph.microsoft.com/v1.0/sites/"+$SharePointTenant+":/sites/"+$SiteName+"?$select=id" $RestData1 = Invoke-MgGraphRequest -Method GET -Uri $uri -ContentType "application/json" # Display Site ID $RestData1.displayName $RestData1.id  But now I want to perform the same operation as above but using msgraph library. The problem is I am not sure how to do it. So far I have: import asyncio from azure.identity.aio import ClientSecretCredential from msgraph import GraphServiceClient tenant_id="..." client_id="..." client_secret="..." credential = ClientSecretCredential(tenant_id, client_id, client_secret) scopes = ['https://graph.microsoft.com/.default'] client = GraphServiceClient(credentials=credential, scopes=scopes) sharepoint_tenant = "foobar365.sharepoint.com" site_name = "someTestSPsite" site_id = "f482d14d-6bd1-1234-gega-3121183eb87a" # how do I get this using GraphServiceClient from the site_name ? # so that I can do this # this works with the id retrieved from PS script drives = (asyncio.run(client.sites.by_site_id(site_id).drives.get())) print(drives) 

Answer: To get the Site ID from Site name and get Drives, modify the code like below: import asyncio from azure.identity.aio import ClientSecretCredential from msgraph import GraphServiceClient from msgraph.generated.sites.sites_request_builder import SitesRequestBuilder from kiota_abstractions.base_request_configuration import RequestConfiguration tenant_id = "TenantID" client_id = "ClientID" client_secret = "Secret" credential = ClientSecretCredential(tenant_id, client_id, client_secret) scopes = ['https://graph.microsoft.com/.default'] graph_client = GraphServiceClient(credential, scopes) query_params = SitesRequestBuilder.SitesRequestBuilderGetQueryParameters( search="RukSite",) request_configuration = RequestConfiguration( query_parameters=query_params, ) async def get_site_id(): result = await graph_client.sites.get(request_configuration=request_configuration) site_id = result.value[0].id return site_id # Get the drives of the site by site_id async def get_drives(site_id): drives = await graph_client.sites.by_site_id(site_id).drives.get() return drives async def main(): site_id = await get_site_id() # Print the Site ID print(f"Site ID: {site_id}") # Fetch the drives for the site using the site ID drives = await get_drives(site_id) # Print the drives print(f"Drives: {drives}") # Run the main async function asyncio.run(main())   Make sure to assign Sites.Read.All application type API permission.
Question: I'm starting to use uv for my CI as it's showing outstanding performances with respect to normal pip installation. for each CI run I (in fact nox does it on my behalf) create a virtual environment that will be used to run the tests. In this environment I run the following: uv pip install .[test]  my folder is a simple python package as this one: my_package/ ├── __init__.py └── logic.py docs/ ├── index.rst └── conf.py test/ └── test_something.py pyproject.toml noxfile.py  As uv is caching everything my virtual environment is never updated and I cannot check new fonctionalities without rebuilding the venv from scratch. how could I make sure that "." get's reinstalled from source everytime I run my tests ? I see 3 potential options:  always install in editable mode: uv pip install -e .[test] which is not ideal for testing purposes as I don't check if the wheel build includes all the necessary files. force the reinstall in the CI call: uv pip install --reinstall .[test] I guess I will loose the caching for all the libs and not only my package force the reinstall from the pyproject.toml: reinstall-package = ["."] but I don't know if it's going to mess with normal installation from users that are not running the tests  Am I missing an alternative and which one is the best to avoid unwanted side effects in my tests ?

Answer: The uv team has been super fast to answer my question and since https://github.com/astral-sh/uv/issues/12038 it's part of the default mechanism:  local sources are now always reinstalled
Question: I used the socket library in Python, and made two simple projects which are server.py and client.py, then I executed them as server.exe and client.exe When I run both (server.exe and client.exe) on my computer, it works fine, but when I run them on two separate computers, the client can not connect to the server that runs on the other computer. Here is my code: Server side: import socket server = socket.socket(socket.AF_INET,socket.SOCK_STREAM) # use our ipv4 for making server on port 9000, and show it on the screen ip = socket.gethostbyname(socket.gethostname()) server.bind((ip,9000)) server.listen(1) print(f"The server ipv4 , give it to client to connect to the server -> {ip}") print("Waiting for client ...") client_socket = server.accept()[0] client_socket.send(b"hello") client_socket.close() server.close() print("Done !!")  Client side: import socket server_ip = input("Please enter server ipv4 here : ") client = socket.socket(socket.AF_INET,socket.SOCK_STREAM) client.connect((server_ip,9000)) # receive data from server (b"hello" which sent from the server) Print(client.recv(5).decode('utf-8')) client.close() print("Done !!")  There are no more exceptions in these two codes but just one exception appears on the client side when I am trying to connect the client to the server which runs on another computer as I said above. The exception on the client side says that the client socket attempted to connect to the server, but it could not connect to the server, and it said something about timeout connection. I searched a lot and watched lots of videos that make the same projects for testing sockets in python and it works for them, but not for me :( I allowed the client and server in firewall on two computers, also disabled the firewall, but it did not work at all, and the same exception appeared. I used netstat -a (cmd command) on the computer on which the server was running on it, and saw that the server socket was listening on port 9000. Nothing seems to work, any help will be appreciated.

Answer: I think this is virtually impossible without an online host or any third party programs, if I say correctly, this because a main difference between network and internet, we use network for devices which connected to a same router, these devices can easily communicating together because they are in a local network, but we use internet for groups of devices (network groups) connection through different routers. This is what I understood after researches.(sorry if I explained wrongly) Are there any free host for testing projects (host for python which supports socket libraries) ?
Question: I am trying to copy data from an iceberg table to a postgres table using a glue job. I have this code: def execute_job(spark, factory: DependencyFactory, environment, logger): print("Starting job") sql = f"SELECT * FROM {DATABASE_NAME}.transactions ORDER BY unique_identifier, in_date" df = spark.sql(sql) df.show(10) config = Config(environment) window_spec = Window.partitionBy("unique_identifier").orderBy("in_date") df = df.withColumn("version", row_number().over(window_spec)) df = df.withColumn("start_date", col("in_date")) df = df.withColumn("end_date", lit("9999-12-31").cast("string")) df.show(10) df = df.withColumn("prev_start_date", lag("start_date").over(window_spec)) df = df.withColumn("end_date", when(col("version") == 1, col("end_date")).otherwise(col("prev_start_date"))) df = df.withColumn("id", col("unique_identifier").cast("string") + ":" + col("version").cast("string")) df.show(10) total_count = df.count() print(f"Total records: {total_count}") if total_count == 0: print("No records to insert. Exiting.") return num_partitions = max(1, total_count // 100000) processed_df = df.repartition(num_partitions) print(f"Number of partitions: {processed_df.rdd.getNumPartitions()}") processed_df.show(10) def process_partition(partition): logger.info("Processing partition") print("Processing partition") partition_list = list(partition) insert_query = """ INSERT INTO transaction.transactions_master (start_date, end_date, in_date, out_date, update_time, id, version, unique_identifier, id_tr, alias, update_correlation_id, correlation_id) VALUES %s """ db_client = PostgresDbClient(DbConfigBuilder(config, config['vault']['service_type'], ssl_cert_file=None)) if not partition_list: print("Skipping empty partition") return batch_size = 100000 try: values = [tuple(row) for row in partition_list] db_client.execute_values_query(insert_query, values, page_size=batch_size) print(f"Inserted {len(values)} records successfully") except Exception as e: print(f"Error inserting partition: {e}") try: print("Starting foreachPartition") processed_df.foreachPartition(process_partition) print("Finished foreachPartition") except Exception as e: print(f"Error during foreachPartition: {e}") print("Data successfully copied to transaction.transactions_master PostgreSQL table")  everything seems to be working (it's printing results in the logs, also tested db connection and it seems fine). However, when it gets to processed_df.foreachPartition(process_partition) it's like the code inside process_partition is not getting executed. I'm not seeing anything in the logs. Last entries are just "Starting foreachPartition" and "Data successfully copied to transaction.transactions_master PostgreSQL table" What could be the issue? Am I using foreachPartition wrong?

Answer: foreachPartition function works on executor level, you can see details logs on the executor logs rather than driver logs. Worth checking the executor logs. Also, do you see the data getting updated in PostgreSQL table?
Question: I created a code to insert my dataframe, called df3, into an excel file. My code is working fine, but now I'd like to change background cells color in all cells based on value I tried this solution, I don't get any errors but I also don't get any cells colored My code: def cond_formatting(x): if x == 'OK': return 'background-color: lightgreen' elif x == 'NO': return 'background-color: red' else: return None print(pd.merge(df, df2, left_on='uniquefield', right_on='uniquefield2', how='left').drop('uniquefield2', axis=1)) df3 = df.merge(df2, left_on='uniquefield', right_on='uniquefield2', how='left').drop(['uniquefield2', 'tournament2', 'home2', 'away2', 'result2'], axis=1) df3 = df3[["home","away","scorehome","scoreaway","best_bets","oddtwo","oddthree","htresult","shresult","result","over05ht","over15ht","over05sh","over15sh","over05","over15","over25","over35","over45","goal","esito","tournament","uniquefield"]] df3 = df3.sort_values('best_bets') df3.style.applymap(cond_formatting) # determining the name of the file file_name = camp + '_Last_20' + '.xlsx' # saving the excel df3.to_excel(file_name, freeze_panes=(1, 0)) print('Tournament is written to Excel File successfully.')  How I said, code is working but all background cells color are white (no colors) any suggestion? Thanks for your help

Answer: If you provided complete code, it would be easier for me to set colors for the cells for your excel file. But an example script given below might assist you: import pandas as pd from openpyxl import load_workbook from openpyxl.styles import PatternFill data = { "ID": [1, 2, 3, 4, 5], "Status": ["OK", "NO", "OK", "NO", "OK"] } df = pd.DataFrame(data) excel_filename = "status.xlsx" df.to_excel(excel_filename, index=False) wb = load_workbook(excel_filename) ws = wb.active # Define fill colors green_fill = PatternFill(start_color="00FF00", end_color="00FF00", fill_type="solid") # Green for "OK" red_fill = PatternFill(start_color="FF0000", end_color="FF0000", fill_type="solid") # Red for "NO" # Apply color based on cell value in the "Status" column for row in ws.iter_rows(min_row=2, max_row=ws.max_row, min_col=2, max_col=2): for cell in row: if cell.value == "OK": cell.fill = green_fill elif cell.value == "NO": cell.fill = red_fill wb.save(excel_filename) print("Excel file created successfully!")
Question: I'm making a Dash Multi-Value Dropdown and finding that long labels use a text wrapping process that skips the first line entirely. When I make a Dash Multi-Value Dropdown like this: dcc.Dropdown( options=[ { 'label': str(entity), 'value': entity, 'title': entity, } for entity in entity_list ], multi=True, optionHeight=55, value=entity_list, )  Longer labels, like the 3rd row in this screenshot, use a text wrapping process that skips the first line entirely. The issue is shown in the middle row here:  Is there a way to make those labels wrap text in a way that includes the first line? Even inelegant/hacky solutions are appreciated!

Answer: You can add explicit line breaks (\n) wrapped in html.Pre to preserve the line breaks and white space:  from dash import Dash, html, dcc app = Dash() entity_list = ['vital sign measurements', 'vital sign\nmeasurements'] app.layout = [html.Div( [ dcc.Dropdown( options=[ { 'label': html.Pre(entity), 'value': entity, 'title': entity, } for entity in entity_list ], multi=True, value=entity_list, )], style={"width": "10%"}, ), ] if __name__ == '__main__': app.run()