eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I have generated a set of (x,y,z) coordinates on a unit sphere using the Fibonacci sphere algorithm. Plotted with a 3d scatter plot, they look alright: https://i.imgur.com/OsQo0CC.gif I now want to connect them with edges, i.e. a triangulation. As suggested in How do i draw triangles between the points of a fibonacci sphere? I went for Delaunay triangulation. For that I used the stripy Python package, which provides triangulations on a sphere. First I convert the coords to spherical (degrees) by iterating over the points and using the following formula:  r = float(sqrt(x * x + y * y + z * z)) theta = float(acos(z / r)) # to degrees phi = float(atan2(y, x)) return r, theta, phi  I obtain vertices_spherical, an array of shape (n, 3) where n is the number of points. We don't need the radius, so I discard it, and I have an array of shape (n, 2). Then I convert to radians and build the triangulation, then make a graph out of it:  vertices_lon = np.radians(vertices_spherical.T[0]) vertices_lat = np.radians(vertices_spherical.T[1]) spherical_triangulation = stripy.sTriangulation(lons=vertices_lon, lats=vertices_lat, permute=True) # Build the graph graph: List[Node] = [] for i in range(spherical_triangulation.npoints): node = Node(name=f'{vertices_spherical.T[0][i]}, {vertices_spherical.T[1][i]}', lon=spherical_triangulation.lons[i], lat=spherical_triangulation.lats[i]) graph.append(node) segs = spherical_triangulation.identify_segments() for s1, s2 in segs: graph[s1].add_neighbor(graph[s2]) return graph  (Node is a simple class with a name, lon, lat, and neighbors) I then convert the coordinates back to cartesian, then scatter plot them. For each node, I iterate over its neighbors and draw a line between them. And to my suprise, I get the following result. It seems kind of walnut- or brain-shaped, where there are two hemispheres where the triangulation worked fine, but for some reason the middle is sorta scrunched up along one plane: https://i.imgur.com/AIlLTmS.gif What could be causing this? Is it simply because of some limitation in how triangulation works? Is it because the points on a Fibonacci sphere are not periodical in some way? Or some mistake in my code? Kind of at a loss here, since the conversion to spherical and back seems to work fine, and there are no surprises with the plotting.

Answer: Fixed it! The issue was that my latitudes were between 0 and pi, whereas stripy.sTriangulation expects them to be between -pi/2 and pi/2. If this happens to anyone else, just know that stripy.sTriangulation expects latitudes between -pi/2 and pi/2, and longitudes between 0 and 2pi. So check if that's what you are providing it. Here's an animation showing the fixed sphere: https://imgur.com/JiGyegj.gif
Question: I'm migrating code to polars from pandas. I have time-series data consisting of a timestamp and value column and I need to compute a bunch of features. i.e. from datetime import datetime, timedelta df = pl.DataFrame({ "timestamp": pl.datetime_range( datetime(2017, 1, 1), datetime(2018, 1, 1), timedelta(minutes=15), time_zone="Australia/Sydney", time_unit="ms", eager=True), }) value = np.random.normal(0, 1, len(df)) df = df.with_columns([pl.Series(value).alias("value")])  I need to generate a column containing an indicator if the timestamp is standard or daylight time. I'm currently using map_elements because as far as I can see the isn't a Temporal Expr, i.e. my current code is def dst(timestamp:datetime): return int(timestamp.dst().total_seconds()!=0) df = df.with_columns(pl.struct("timestamp").map_elements(lambda x: dst(**x)).alias("dst"))  (this uses a trick that effectively checks if the tzinfo.dst(dt) offset is zero or not) Is there a (fast) way of doing this using polars expressions rather than (slow) map_elements?

Answer: With polars>=0.18.5 the following works df = df.with_columns((pl.col("timestamp").dt.dst_offset()==0).cast(pl.Int32).alias("dst"))
Question: Learning FastAPI and SQLAlchemy with video tutorial. In video code working correct. Code from tutorial: @router.get("/") async def get_specific_operations(operation_type: str, session: AsyncSession = Depends(get_async_session)): query = select(operation).where(operation.c.type == operation_type) result = await session.execute(query) return result.all()  When I using this code, I have error "Cannot convert dictionary update sequence element #0 to a sequence" I tried: return result.scalars().all()  But I get only id numbers, without another data from strings. With chain: @router.get("/") async def get_specific_operations(operation_type: str, session: AsyncSession = Depends(get_async_session)): query = select(operation).where(operation.c.type == operation_type) result = await session.execute(query) result = list(chain(*result)) return result  I get correct results, but without column titles. This information is not fully understood by the user. I get / I need (this like in video):  / [ [ / { 4 / id: 4 name / username: name 25, / age: 25 7 / }, name2 / { 31 / id: 7 ] / username: name2 / age:31 / } / ]  How I can get full information from string of DB with column names? I need a list of dictionaries, I think.

Answer: I found a solution that helped me. Just changed "return": @router.get("/") async def get_specific_operations(operation_type: str, session: AsyncSession = Depends(get_async_session)): query = select(operation).where(operation.c.type == operation_type) result = await session.execute(query) return [dict(r._mapping) for r in result]
Question: I'm attempting to calculate all possible real X-values at a certain Y-value from a polynomial given in descending coefficent order, in Python 3.10. I want the resulting X-values to be provided to me in a list. I've tried using the roots() function of the numpy library, as shown in one of the answers to this post, however it does not appear to work: import numpy as np import matplotlib.pyplot as plt def main(): coeffs = np.array([1, 2, 2]) y = 1.5 polyDataX = np.linspace(-2, 0) polyDataY = np.empty(shape = len(polyDataX), dtype = float) for i in range(len(polyDataX)): polyDataY[i] = coeffs[0] * pow(polyDataX[i], 2) + coeffs[1] * polyDataX[i] + coeffs[2] coeffs[-1] -= y x = np.roots(coeffs).tolist() plt.axhline(y, color = "orange") plt.plot(polyDataX, polyDataY, color = "blue") plt.title("X = " + str(x)) plt.show() plt.close() plt.clf() if (__name__ == "__main__"): main()  In my example above, I have the coefficents of my polynomial stored in the local variable coeffs, in descending order. I then attempt to gather all the X-values at the Y-value of 0.5, stored within the x and y local variables respectivelly. I then display the gathered X-values as the title of the shown plot. The script above results in the following plot:  With the X-values being shown as [-2.0, 0.0], instead of the correct:  What is the proper way to get all real X-values of a polynomial at a certain Y-value in Python? Thanks for reading my post, any guidance is appreciated.

Answer: You should be making use of the numpy.polynomial.Polynomial class that was added in numpy v1.4 (more information here). With that class, you can create a polynomial object. To find your solution, you can subtract y from the Polynomial object and then call the roots method. Another nice feature is that you can directly call the object, which you can use to compute polyDataY. Just note that the Polynomial class expects the coefficients to be given backward from np.roots, i.e. a quadratic of x^2 + 2x + 2 should have the coefficients (2, 2, 1). To keep things consistent with what you gave, I just pass the reversed coeffs. import numpy as np from numpy.polynomial import Polynomial import matplotlib.pyplot as plt plt.close("all") coeffs = np.array([1, 2, 2]) poly = Polynomial(coeffs[::-1]) polyDataX = np.linspace(-2, 0) polyDataY = poly(polyDataX) y = 1.5 x = (poly - y).roots() plt.axhline(y, color = "orange") plt.plot(polyDataX, polyDataY, color = "blue") plt.title("X = " + str(x)) plt.show()
Question: I want to replicate rows of a dataframe as to prepare for the adding of a column. The dataframe contains years column and I want to add a fixed column of months. The idea is to replicate each same year rows exactly 12 times then add a fixed value column (1-12). my code is the following:  all_years = dataframe["Year"].unique().tolist() new_dataset = pd.DataFrame() for idx, year in enumerate(all_years): rows_dataframe = pd.concat( [dataframe.where(dataframe["Year"] == year).dropna()] * 12, ignore_index=True) new_dataset = pd.concat([rows_dataframe, new_dataset], ignore_index=True)  The results are correct, but can I avoid the for loop here, and implement this in a more "pandas-ic" way? EDIT: expected results for one value of years (here 2012) is: (to note that months column is not added through my code, but added it to show the final output) +-------+--------+---------+ | Years | Months | SomeCol | +-------+--------+---------+ | 2011 | 12 | val1 | +-------+--------+---------+ | 2012 | 1 | val1 | +-------+--------+---------+ | 2012 | 2 | val1 | +-------+--------+---------+ | 2012 | 3 | val1 | +-------+--------+---------+ | 2012 | 4 | val1 | +-------+--------+---------+ | 2012 | 5 | ... | +-------+--------+---------+ | 2012 | 6 | ... | +-------+--------+---------+ | 2012 | 7 | val1 | +-------+--------+---------+ | 2012 | 8 | val1 | +-------+--------+---------+ | 2012 | 9 | val1 | +-------+--------+---------+ | 2012 | 10 | | +-------+--------+---------+ | 2012 | 11 | | +-------+--------+---------+ | 2012 | 12 | | +-------+--------+---------+ | 2013 | 1 | ... | +-------+--------+---------+ 

Answer: How about using groupby. I am assuming each year occurs only once in your data frame: new_dataset = dataframe.groupby("Year").apply(lambda x: pd.concat([x] * 12)).reset_index(drop=True)  Or just repeat and sort the values: new_dataset = pd.concat([dataframe] * 12).sort_values("Year").reset_index(drop=True)
Question: In the current path there are multiple folders, each folder has multiple folders or xlsx files inside,I want to iterate through each folder and read the xlsx files until there are no more folders or until all xlsx files are read. There are 50+ folders and 2000+ excel files. Below is my code: import os import pandas as pd current_path=os.getcwd() dfs = [] def process_folder(path): for item in os.listdir(path): item_path=os.path.join(path, item) if os.path.isdir(item_path): process_folder(item_path) elif item.endswith('.xlsx'): df = pd.read_excel(item_path) dfs.append(df) process_folder(current_path) result_df = pd.concat(dfs, ignore_index=True) result_df.to_excel('result.xlsx')  when I run the code, it shows error:"Excel file cannot be determined, you must specify an engine manually". So I modified read_excel: df = pd.read_excel(item_path, engine='openpyxl'). Then there is the error: "zipfile.BadZipFile: File is not a zip file." However, I didn't read any zipfile. Not sure why this error shows up.

Answer: You probably have files with the extension .xlsx but which are not real Excel files. To find them, you can use: import pathlib for filename in pathlib.Path.cwd().glob('*.xlsx'): with open(filename, 'rb') as xlsx: sig = xlsx.read(2) if sig != b'PK': print(f'"{filename}" does not appear to be a valid Excel file')  Testing the signature of a zip file with .xlsx extension should be sufficient for the moment.
Question: New at pandas, having an issue relating two DataFrames together based on the function of two columns in each DataFrame. I have two DataFrames - one representing road segments, and another with points somewhere near that road. Road Shape DataFrame  shape_pt_lat shape_pt_lon shape_pt_sequence 2583910 53.402329 -6.150988 1 2583911 53.402334 -6.151043 2 2583912 53.402345 -6.151175 3 2583913 53.402359 -6.151328 4 2583914 53.402518 -6.152953 5 ... ... ... ...  Points DataFrame  latitude longitude timestamp 0 53.376873 -6.216212 1.686826e+09 1 53.370968 -6.223517 1.686827e+09 2 53.363358 -6.234719 1.686827e+09 3 53.360840 -6.238742 1.686827e+09 4 53.355160 -6.246171 1.686827e+09 .. ... ... ...  I want to add a column to the Points DataFrame that contains the index of the Road DataFrame row containing the closest point. I can do this by iterating through the Points DataFrame, and for each row then iterating through the Road DataFrame, calculating each distance, and finding a minimum, but this seems very inefficient. Is there a better way to achieve this? EDIT I tried to adapt the approach suggested, it works but it ends up taking about 4x as long as simply doing all the iterations. Is there any way to make this faster? shape_tup = [tuple(r) for r in shape_df[['shape_pt_lat', 'shape_pt_lon']].to_numpy()] pos_points["shape_pt_ind"] = np.nan for index, pos in pos_points.iterrows(): min_pair = min(shape_tup, key=lambda t: (abs(t[0] - pos["latitude"]) + abs(t[1] - pos["longitude"]))) min_index = shape_df.index[(shape_df[['shape_pt_lat', 'shape_pt_lon']].values[:, None] == min_pair).all(2).any(1)] pos_points.loc[index, "shape_pt_ind"] = min_index[0] 

Answer: The simplest approach is to use the cdist function from scipy. It calculates the distance between every point in points and every point in road. You have the choice of 20 or so distance functions. After that it's trivial to find the closest point: from scipy.spatial.distance import cdist distance = cdist( points[["latitude", "longitude"]], road[["shape_pt_lat", "shape_pt_lon"]], metric="euclidean" ) points["shape_pt_sequence"] = road["shape_pt_sequence"].to_numpy()[distance.argmin(axis=1)]  If your coordinates are far apart, the geodesic distance is more appropriate: from scipy.spatial.distance import cdist from geopy.distance import geodesic distance = cdist( points[["latitude", "longitude"]], road[["shape_pt_lat", "shape_pt_lon"]], metric=lambda a, b: geodesic(a, b).kilometers, ) points["shape_pt_sequence"] = road["shape_pt_sequence"].to_numpy()[distance.argmin(axis=1)]
Question: Here I have a simple code that is not seeming to do what I am trying to do. The goal of this code is to retrieve the value of the stock that the user inputs. ticker = input("What stock would you like to track?: (enter the ticker) ") r = requests.get(f"https://www.marketwatch.com/investing/stock/{ticker}?mod=search_symbol").text webpage = BeautifulSoup(r, 'html.parser') locate = webpage.find('bq-quote', {'class': 'value'}) print(locate)  Although the value of the stock is inside the class value it does not seem to be returning anything. web.find_all() just returns '[]' so I am not sure what I am doing wrong here.

Answer: This works. from bs4 import BeautifulSoup import requests ticker = input("What stock would you like to track?: (enter the ticker) ") r = requests.get(f"https://www.marketwatch.com/investing/stock/{ticker}?mod=search_symbol",).text webpage = BeautifulSoup(r, "html.parser") locate = webpage.find(class_="value") stock_price = locate.text print(stock_price)
Question: I am trying to upload my package to PyPi. It uploads successfully every time but when I try to install it, I get the following error: ERROR: Could not find a version that satisfies the requirement Google-Ads-Transparency-Scraper==1.4 (from versions: none) ERROR: No matching distribution found for Google-Ads-Transparency-Scraper==1.4  The following is the directory structure I have   GoogleAds  GoogleAdsTransparency  __init __.py main.py regions.py   setup.py setup.cfg license.txt README.md     The setup.py has the following content """Install packages as defined in this file into the Python environment.""" from setuptools import setup, find_packages setup( name="Google Ads Transparency Scraper", author="Farhan Ahmed", author_email="jattfarhan10@gmail.com", url="https://github.com/faniAhmed/GoogleAdsTransparencyScraper", description="A scraper for getting Ads from Google Transparency", version="1.4", packages=find_packages(), download_url= 'https://github.com/faniAhmed/GoogleAdsTransparencyScraper/archive/refs/tags/v1.2.tar.gz', keywords= ['Google', 'Transparency', 'Scraper', 'API', 'Google Ads', 'Ads', 'Google Transparency', 'Google Transparency Scraper', 'Google Ads Scrapre'], license='Securely Incorporation', install_requires=[ "setuptools>=45.0", "beautifulsoup4>=4.12.2", "Requests>=2.31.0", "lxml>=4.6.3", ], classifiers=[ "Development Status :: 5 - Production/Stable", "Environment :: Console", "Intended Audience :: Developers", "License :: Free for non-commercial use", "Natural Language :: Urdu", "Operating System :: OS Independent", "Programming Language :: Python", ], platforms=["any"], )  The setup.cfg [metadata] description-file = README.md  The init.py has the following content from GoogleAdsTransparency.main import GoogleAds from GoogleAdsTransparency.main import show_regions_list  I run the following commands to upload the package python setup.py sdist twine upload dist/*  The link to the package is https://pypi.org/project/Google-Ads-Transparency-Scraper/

Answer: This might be your problem, The file uploaded contains spaces where as when you see the second image, it should not contain spaces   By examining that, I come to the conclusion as to what is wrong. Do you have spaces inside your setup.py? If so, remove them, also the github page for your project is incomplete, it does not contain setup.py, so I can't diagnose any further. Please, if possible, give out the complete source code for such highly specific problems. To look at the downloads available for a package:  So, i dug a little deeper, and found out this:  Your name contains spaces and your download_url is outdated. After digging deeper by looking at verbose logs of pip:  I find that it is able to find the download link for the packages, but the issue is that, it is unable to find the project version from the link.
Question: I have a DataFrame which contains a date column. I need to get the unique values of that column and then sort them out. I started with: unique_dates = data['date'].unique()  Which works well. Next, I need to sort this unique list of dates. However, I am having issues because unique_values is now of DatetimeArray type, and this does not have the sort() or sort_values() methods I can find in a DataFrame. If I could convert it to just a NumPy array, I could sort it, but I am not finding a way to do that either... How can I get a DatetimeArray sorted? Is there another way of doing this that I am not considering? Thanks! Eduardo

Answer: The DatetimeArray is a flavour of ExtensionArray. ExtensionArrays don't have a sort or sort_values method, but they do have argsort, so we can achieve sorting using: unique_dates_sorted = unique_dates[unique_dates.argsort()]  i.e. we get the indexes which result in the sorted array, then use those to produce a reordered array. A complete mini-example: import datetime import pandas as pd dates = [datetime.date(2021, 1, 1)] * 50 + [datetime.date(2022, 1, 1)] * 50 + [datetime.date(2020, 1, 1)] * 50 df = pd.DataFrame(dates, columns=['date']) unique_dates = df['date'].unique() unique_dates_sorted = unique_dates[unique_dates.argsort()] print(unique_dates_sorted)  giving: [datetime.date(2020, 1, 1) datetime.date(2021, 1, 1) datetime.date(2022, 1, 1)]
Question: Could you tell me please, if I use for i in range(n), does python creates a range of 0 .. n-1, and iterates elements in this container (O(n) extra space), or it use only 1 vatiable (O(1)) space) From one side, I thought, if we can convert range to list, then using range function creates a container (O(n)).but from the other side, instead of using for i in range(n), we can use while i < n ... O(1).

Answer: In Python 3, range does not create all the elements, but only returns the current element when you request it. So yes, it's using O(1) space.
Question: I wanted to maximize gross profit margin (total profit/total revenue) with binary variables, say whether products will be in the mix or not by that variables will be 1 or 0 (binary), trying to solve with gekko mixed integer non linear programming here is the example for 3 products, where we want to keep any 2 products variables for 3 products, x1, x2 and x3 total profit = 150*x1 + 120*x2 + 100*x3 total revenue = 200*x1 + 150x2 + 250*x3 gross profit margin = total profit/ total revenue solution tried m = GEKKO() x1 = m.Var(integer=True, lb=0, ub=1) x2 = m.Var(integer=True, lb=0, ub=1) x3 = m.Var(integer=True, lb=0, ub=1) m.Maximize((150*x1 + 120*x2 + 100*x3)/(200*x1 + 150*x2 + 250*x3)) m.Equation(x1 + x2 + x3 == 2) m.options.SOLVER = 1 m.solve()  result x1: 0 x2: 0 x3: 0 objective function: nan things tried i) tried with adding one more constraint with the denominator > 0, getting same solution ii) tried with changing lb=0 to any other integer value and it is working (say lb=1, ub=2), not sure if anything particularly needs to be added for using lb=0 ii) tried absolute profit maximization (removing the denominator) and it is working fine any help will be appreciated, thanks in advance

Answer: Try using a different initial guess than 0 for at least one of the variables: x1 = m.Var(value=1,integer=True, lb=0, ub=1)  Gekko uses a default starting value of 0. Gradient-based optimizers need to calculate a search direction and the initial guesses of zero lead to an Inf evaluation of the objective function. Here is the complete script with a condensed way to define the variables: from gekko import GEKKO m = GEKKO() x1,x2,x3 = m.Array(m.Var,3,value=1,integer=True,lb=0,ub=1) m.Maximize((150*x1 + 120*x2 + 100*x3)/(200*x1 + 150*x2 + 250*x3)) m.Equation(x1 + x2 + x3 == 2) m.options.SOLVER = 1 m.solve()  This gives a successful solution:  --------------------------------------------------- Solver : APOPT (v1.0) Solution time : 3.579999999965366E-002 sec Objective : -0.771428571428571 Successful solution ---------------------------------------------------  Gekko converts maximization problems to minimization problems that are standard for MINLP and NLP solvers so the objective function value is the negative of the true value. Use this to get the Maximization objective of 0.7714: print(-m.options.OBJFCNVAL)
Question: when plotting data using plotly the y-zero line is not shown in the grid (it is blank), creating a weird blank space. I've tried to modify the grid parameters so y-zero and x-zero lines are always displayed, but it hasn't work. I attach the function that plots the data. See the image and how for y=0 there is no grey line: example of the behaviour def plot_dictionary_lines(data_dict, title, colors, dtick_y, y_title): tilt_values = list(data_dict.keys()) orientation_values = list(data_dict[tilt_values[0]].keys()) fig = go.Figure() for i,tilt in enumerate(tilt_values): tilt_data = data_dict[tilt] y_values = [] for orientation in orientation_values: y_values.append(tilt_data[orientation]) fig.add_trace(go.Scatter(x=orientation_values, y=y_values, mode='lines', name=tilt, line=dict(color=colors[i]))) fig.update_layout(title=title, xaxis_title='Orientation') fig.update_layout(yaxis=dict(title=y_title, dtick=dtick_y, minor=dict(ticks="inside", showgrid=True))) fig.update_layout(title=title, title_font_size=12, title_x=0.5, margin=dict(l=5, r=5, t=30, b=5)) fig.update_layout(showlegend=True, plot_bgcolor= 'white' ) fig.update_xaxes(showline=True, mirror=True, linewidth=2, linecolor='black', gridcolor='grey', gridwidth=1) fig.update_yaxes(showline=True, mirror=True, linewidth=2, linecolor='black', gridcolor='grey', gridwidth=1) fig.update_layout(legend_title_text='Tilt (deg)') fig.show()  Data to reproduce the plot: data_dict = {'20': {'0': -29.993976613550426, '90': -9.982751500722712, '180': 0.0, '270': -16.409959148158066}, '40': {'0': -51.454747907503716, '90': -13.263266344556737, '180': 0.0, '270': -23.372002876678337}, '60': {'0': -60.76495332385845, '90': -13.362989512932888, '180': 0.0, '270': -25.197913842540597}} title = "Solar production sold entirely, respect south" color_codes = { "10colors": [ '#cce542', '#98d956', '#66cb69', '#2fba79', '#00a884', '#009589', '#008286', '#006e7d', '#185b6d', '#2a4858'] } colors = color_codes["10colors"] d_ticky = 5 y_title = "%" 

Answer: That took me forever. Apparently the zero line does NOT follow the grid color. It has separate properties, zerolinecolor and zerolinewidth. Yours is currently white on a white background. Change it like this (or another color): fig.update_layout(yaxis=dict(zerolinecolor='black'))