eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: Context: When running a regex match in Perl, $1, $2 can be used as references to captured regex references from the match, similarly in Python \g<0>,\g<1> can be used Perl also has a $+ special reference which refers to the captured group with highest numerical value My question: Does Python have an equivalent of $+ ? I tried \g<+> and tried looking in the documentation which only says:  There’s also a syntax for referring to named groups as defined by the (?P<name>...) syntax. \g<name> will use the substring matched by the group named name, and \g<number> uses the corresponding group number. \g<2> is therefore equivalent to \2, but isn’t ambiguous in a replacement string such as \g<2>0. (\20 would be interpreted as a reference to group 20, not a reference to group 2 followed by the literal character '0'.) The following substitutions are all equivalent, but use all three variations of the replacement string. 

Answer: The method captures in the regex module provides the same functionality: it "returns a list of all the captures of a group." So get the last one >>> import regex >>> str = 'fza' >>> m = regex.search(r'(a)|(f)', str) >>> print(m.captures()[-1]) f  When the str has a before f this code prints a. This is the exact equivalent of Per's $+. Here we do get all captures, not only the highest one, and other related methods. Follow the word "captures" in the linked docs.  Another option that fits the intended use, explained in a comment, is the branch reset group, (?|pattern). It is also available in the regex module. >>> import regex >>> m = regex.search(r'(?|(a)|(b))', 'zba') >>> m.group(1) 'b'  In short, with the branch reset (?|(pA)|(pB)|(pC)) the whole pattern is one capture group (with three alternations), not three. So you always know which is the "last" capture as there is only one, which has the match. This can be used with named capture groups as well. This feature adds far more power as the pattern in (?|...) gets more complex. Find it in your favorite regex documentation. Here it is in regular-expressions.info, for example, and here are some Perl resources, in perlre and an article in The Effective Perler.
Question: I have create a webscraper that sends notifications to my phone whenever certain events are detected. So far I have achieved this by sending emails through the sendgrid api. Its a pretty nice service, and it is free, but it clutters up the mailbox quite a bit. In stead I’d like to send messages directly to the iOS notification bar. Does anyone here has experience with sending push-notifications to iOS and can point me in the correct direction? I would be happy with a subscription service, but would off course prefer a solution that does not require a third party if it is possible. I have tested PushNotifier, but I found it a bit clunky, and the notifications are neither customisable or beautiful. Its also not a free service, which would have been a great plus.

Answer: Maybe you should check out pushover.net. They have a simple WebAPI to send customized notifications to iOS devices. See https://support.pushover.net/i44-example-code-and-pushover-libraries#python for code samples.
Question: I'm visualizing a scatterplots with between 400K and 2.5M points. I expectected to need to downsample before visualizing but to see just how much I ran a pilot test with a 400k dataset in plotly express, and the plot popped up quickly, beautifully, and responsively. In order to make the interractive figure I really need to use plotly.graph_objects, as I need multiple traces with different colorscales, so I made basically the same graph with graph_objects and it wasn't just slower, it crashed my computer. I'd really like to downsample as little as possible and I'm surprised by the sheer performance difference between these two approaches so I guess that boils down to my question: Why is there such a performance difference and is it possible to change layout/figure/whatever parameters in graph_objects so to close the gap? Here is a snippet to show what I mean by basically the same graph: graph_objects  fig = go.Figure() fig.add_trace(go.Scatter(x = x_values, y = y_values, opacity = opacity, marker = { 'size': size, 'color': community, 'colorscale': colorscale }))  express  pacmap_map = px.scatter(x = x_values, y = y_values, color_continuous_scale=colorscale, opacity = opacity, color = community) pacmap_map.update_traces(marker = { 'size': size })  I would have expected performance to either be identical or at least in the same ballpark, but express works like a dream and graph_objects crashes the jupyter kernel and whatever IDE it is running from, so a large difference.

Answer: Running the following simple example: import numpy as np import plotly.graph_objects as go import plotly.express as px x = np.linspace(-2, 2, 100000) y = np.cos(x) fig = go.Figure(data=[go.Scatter(x=x, y=y)]) fig2 = px.scatter(x=x, y=y) type(fig.data[0]), type(fig2.data[0]) # out: (plotly.graph_objs._scatter.Scatter, plotly.graph_objs._scattergl.Scattergl)  As you can see, plotly express appears to switch to Scattergl when the number of points is higher than some threshold. Scattergl renders on an html5 canvas, hence it uses the GPU (hence efficiency). Whereas Scatter creates svg objects that get inserted in the current document, consuming muuuuuch more memory.
Question: I use Python and SPARQL to make a scheduled query for a database. I tried to use the python f-string and doc-string to inject today's date in the query, but when I try so, a conflict occurs with SPARQL syntax and the python string. The better way would be to use SPARQL to get today's date. In my python file my query looks like this: query = """ PREFIX skos: <http://www.w3.org/2004/02/skos/core#> PREFIX xsd: <http://www.w3.org/2001/XMLSchema#> SELECT * { VALUES (?currentDateString) {(today)} FILTER(xsd:date(?dateApplicabilityNode) >= xsd:date(?validDateString) && xsd:date(?dateApplicabilityNode) <= xsd:date(?currentDateString)) } GROUP BY ... ORDER BY ...  How to get today's date in the format of "YYYY-MM-DD"?

Answer: now() returns the datetime (as xsd:dateTime) of the query execution: BIND( now() AS ?currentDateTime ) .  To get only the date (as xsd:string), you could use CONCAT() with year(), month(), and day(): BIND( CONCAT( year(?currentDateTime), "-", month(?currentDateTime), "-", day(?currentDateTime) ) AS ?currentDateString ) .  (To get the date as xsd:date, you could use xsd:date(?currentDateString).)
Question: I want to configure my pip using environmental variables. I already have two pip index urls. So I'm already using PIP_INDEX_URL and PIP_EXTRA_INDEX_URL variables. PIP_INDEX_URL="https://example.com" PIP_EXTRA_INDEX_URL="https://example2.com"  But I want to add one more index url. I don't know how I tried to add it with ; PIP_INDEX_URL="https://example.com" PIP_EXTRA_INDEX_URL="https://example2.com;https://example3.com"  But it didn't seem to work

Answer: Pip expects an empty space ( ) to separate the values in environment variables. In this case, for example: PIP_EXTRA_INDEX_URL="https://example2.com https://example3.com"  See pip's documentation section "Environment variables".
Question: I have an image file in avif format How can I convert this file to png format? I found some code to convert jpg files to avif, but I didn't find any code to reconvert them.

Answer: You need to install this modules: pip install pillow-avif-plugin Pillow Then: from PIL import Image import pillow_avif img = Image.open('input.avif') img.save('output.png')
Question: I'm trying to move from Selenium to Playwright for some webscraping tasks. Perhaps I got stuck into this bad habit of having Selenium running the browser on the side while testing the commands and selectors on the run. Is there any way to achieve something similar using Playwright? What I achieved so far was running playwright on the console, something similar to this: from playwright.sync_api import sync_playwright with sync_playwright() as pw: browser = pw.chromium.launch(headless=False) page = browser.new_page() page.goto('https://google.com') page.pause()  I get a Browser window together with a Playwright Inspector - from there I none of my commands or variables will execute.

Answer: I'd use the technique from can i run playwright outside of 'with'? and How to start playwright outside 'with' without context managers on the interactive repl: PS C:\Users\foo\Desktop> py Python 3.10.2 (tags/v3.10.2:a58ebcc, Jan 17 2022, 14:12:15) [MSC v.1929 64 bit (AMD64)] on win32 Type "help", "copyright", "credits" or "license" for more information. >>> from playwright.sync_api import sync_playwright >>> p = sync_playwright().start() >>> browser = p.chromium.launch(headless=False) >>> page = browser.new_page() >>> page.goto("https://www.example.com") <Response url='https://www.example.com/' request=<Request url='https://www.example.com/' method='GET'>> >>> page.title() 'Example Domain' >>> page.close() >>> browser.close() >>> p.stop()  If you use page.pause(), try running playwright.resume() in the browser dev tools console to resume the Python repl. If you really need to do this from a script rather than the Python repl, you could use the code interpreter or roll your own, but I'd try to avoid this if possible.
Question: So I try to enhance this image by applying log transform on it original image The area where there are bright white color turns into color blue on the enhanced image. enhanced image path = '...JPG' image = cv2.imread(path) c = 255 / np.log(1 + np.max(image)) log_image = c * (np.log(image + 1)) # Specify the data type so that # float value will be converted to int log_image = np.array(log_image, dtype = np.uint8) cv2.imwrite('img.JPG', log_image)  There's also a warning: RuntimeWarning: divide by zero encountered in log I tried using other type of log (e.g log2, log10...) but it still show the same result. I tried changing dtype = np.uint32 but it causes error.

Answer: Same cause for the two problems Namely this line log_image = c * (np.log(image + 1))  image+1 is an array of np.uint8, as image is. But if there are 255 components in image, then image+1 overflows. 256 are turned into 0. Which lead to np.log(imag+1) to be log(0) at this points. Hence the error. And hence the fact that brightest parts have strange colors, since they are the ones containing 255 So, since log will have to work with floats anyway, just convert to float yourself before calling log path = '...JPG' image = cv2.imread(path) c = 255 / np.log(1 + np.max(image)) log_image = c * (np.log(image.astype(float) + 1)) # Specify the data type so that # float value will be converted to int log_image = np.array(log_image, dtype = np.uint8) cv2.imwrite('img.JPG', log_image)
Question: Problem: I want to calculate at several times the adjacency matrix A_ij given the adjacency list E_ij, where E_ij[t,i] = j gives the edge from i to j at time t. I can do it with the following code: import numpy as np nTimes = 100 nParticles = 10 A_ij = np.full((nTimes, nParticles, nParticles), False) E_ij = np.random.randint(0, 9, (100, 10)) for t in range(nTimes): for i in range(nParticles): A_ij[t, i, E_ij[t,i]] = True  Question: How can I do it in a vectorized way, either with fancy indexing or using numpy functions such as np.take_along_axis?  What I tried: I expected this to work: A_ij[:,np.arange(nParticles)[None,:,None], E_ij[:,None,np.arange(nParticles)]] = True  But it does not.  Related to: Trying to convert adjacency list to adjacency matrix in Python

Answer: I think this might work: import numpy as np nTimes = 100 nParticles = 10 A_ij = np.full((nTimes, nParticles, nParticles), False) E_ij = np.random.randint(0, 9, (100, 10)) np.put_along_axis(A_ij, E_ij[..., None], True, axis=2)
Question: My dataframe is in weekly level as below:  I was trying to implement a prophet model using this code: df.columns = ['ds', 'y'] # define the model model = Prophet(seasonality_mode='multiplicative') # fit the model model1 = model.fit(df) model1.predict(10)  I need to predict the output in a weekly level for the next 10 weeks. How can I fix this?

Answer: You need to use model.make_future_dataframe to create new dates: model = Prophet() model.fit(df) future = model.make_future_dataframe(periods=10, freq='W') predictions = model.predict(future)  predictions will give predicted values for the whole dataframe, you can reach to the forecasted values for the next 10 weeks with simple indexing: future_preds = predictions.iloc[-10:]
Question: I need help, please be kind I'm a beginner. I have a nested dict like this: dict_ = { "timestamp": "2022-11-18T10: 10: 49.301Z", "name" : "example", "person":{ "birthyear": "2002" "birthname": "Examply" }, "order":{ "orderId": "1234" "ordername": "onetwothreefour" } }  How do I get a new dict like: new_dict = {"timestamp": "2022-11-18T10: 10: 49.301Z", "birthyear": "2002", "birthname": "Examply", "orderId": "1234"}  I tried the normal things I could google. But I only found solutions like getting the values without the keys back or it only works for flatten dicts. Last thing I tried: new_dict = {key: msg[key] for key in msg.keys() & {'timestamp', 'birthyear', 'birthname', 'orderId'}  This do not work for the nested dict. May someone has an easy option for it.

Answer: A general approach: dict_ = { "timestamp": "2022-11-18T10: 10: 49.301Z", "name": "example", "person": { "birthyear": "2002", "birthname": "Examply" }, "order": { "orderId": "1234", "ordername": "onetwothreefour" } } def nested_getitem(d, keys): current = d for key in keys: current = current[key] return current new_dict = {"timestamp": nested_getitem(dict_, ["timestamp"]), "birthyear": nested_getitem(dict_, ["person", "birthyear"]), "birthname": nested_getitem(dict_, ["person", "birthname"]), "orderId": nested_getitem(dict_, ["order", "orderId"]), } print(new_dict)  Output {'timestamp': '2022-11-18T10: 10: 49.301Z', 'birthyear': '2002', 'birthname': 'Examply', 'orderId': '1234'}
Question: I've a x and y. Both are flattened 2D arrays. I've two similar arrays, one for determining the colour of datapoint, another for determining detection method ("transit" or "radial"), which is used for determining the marker shape.  a=np.random.uniform(0,100,(10,10)).ravel() #My x b=np.random.uniform(0,100,(10,10)).ravel() #My y d=np.random.choice([0,1,2],(10,10)).ravel() #Color map e=np.random.choice(["radial","transit"],(10,10)).ravel() #Marker type  Since there can be only one type of marker in a scatterplot and I have two types of markers, I call the scatterplot twice.  a_radial=a[e=="radial"] b_radial=b[e=="radial"] d_radial=d[e=="radial"] a_transit=a[e=="transit"] b_transit=b[e=="transit"] d_transit=d[e=='transit'] fig,ax=plt.subplots() #One plot each for two types of markers. scatter1=ax.scatter(a_radial,b_radial,c=d_radial,marker='o',label="Radial") scatter2=ax.scatter(a_transit,b_transit,c=d_transit,marker='^',label="Transit") ax.legend(*scatter1.legend_elements(),loc=(1.04, 0),title="Legend") ax.legend(loc=(1.01, 0),title="Detection") plt.show()  This is giving me the following plot  I want a label for color map too but as soon as I add the code for it, the label for "Detection" disappears Why is that and how can I resolve it? I added command for the label legends but it only shows one at a time. I was expecting both of them to show up at the same time.

Answer: You could just manually add the first legend to the Axes: leg1 = ax.legend(*scatter1.legend_elements(), bbox_to_anchor=(1.04, 1), loc="upper left", title="Legend") ax.add_artist(leg1)   However, this is not every clear as the color legend uses the marker for Radial and the Detection legend uses just two arbitrary colors out of the three. So a better solution is to make two neutral legends that don't mix marker symbol and color: import matplotlib.patches as mpatches import matplotlib.lines as mlines # ... handles = [mpatches.Patch(color=line.get_color()) for line in scatter1.legend_elements()[0]] leg1 = ax.legend(handles, scatter1.legend_elements()[1], bbox_to_anchor=(1.04, 1), loc="upper left", title="Legend") ax.add_artist(leg1) handles = [mlines.Line2D([], [], marker=marker, mec='k', mfc='w', ls='') for marker in ['o', '^']] ax.legend(handles, ['Radial', 'Transit'], loc=(1.01, 0),title="Detection")    As an alternative, you could use seaborn's scatterplot, where you can specify hue and style to get two legend categories (although in one legend), see 4th example in the seaborn scatterplot docs. In this example, however, the marker used for the different hues (days) is the same as for Lunch, so I think my solution above is a bit clearer.
Question: I am just started looking/experimenting pyscript as per the current python code which is running on Python 3.6.0. But looks like pyscript loads the python version along with Pyodide and it is retuning the latest stable version based on the Pyodide version. Problem Statement : Is there any way we can change/switch the python version as per the need while working with pyscript ? What did I tried so far to verify the Pyodide and Python version : I checked the version of Pyodide by using below code.   <link rel="stylesheet" href="https://pyscript.net/latest/pyscript.css"/> <script defer src="https://pyscript.net/latest/pyscript.js"></script> <py-script>import pyodide_js; print(pyodide_js.version)</py-script>    As per the above code snippet, It is returning 0.21.2, Now to check python version.   <link rel="stylesheet" href="https://pyscript.net/latest/pyscript.css"/> <script defer src="https://pyscript.net/latest/pyscript.js"></script> <py-script> import pyodide_js; import sys; print('Pyodide version : ' + pyodide_js.version) print('Python version : ' + sys.version) </py-script>    It is returning 3.10.2 but I want to change/switch it to 3.6.0.

Answer: YOu cannot as Python is built into Pyodide. You would need to rebuild Pyodide to change the version of Python. I also do not think that Python 3.6 will work with the current version of PyScript and Pyodide. Your only practical option is to make your application work with the Pyodide version of Python.
Question: Based on questions like this What makes C faster than Python? I've learned that dynamic/static typing isn't the main reason that C is faster than Python. It appears to be largely because python programs are interpreted, and c programs are compiled. I'm wondering if strict typing would close the gap in performance for interpreted vs compiled programs enough that strict typing would be a viable strategy for improving interpreted Python program performance post facto? If the answer is yes, is this done in pro-dev contexts?

Answer: With current versions of Python, type annotations are mostly hints for the programmer and possibly some validation tools but are ignored by the compiler and not used at runtime by the byte-code interpreter, which is similar to the behavior of Typescript. It might be possible to change the semantics of Python to take advantage of static typing in some circumstances to generate more efficient byte-code and possibly perform just in time executable code generation (JIT). Advanced Javascript engines use complex heuristics to achieve this without type annotations. Both approaches could help make Python programs much faster and in some cases perform better than equivalent C code. Note also that many advanced Python packages use native code, written in C and other languages, taking advantage of optimizing compilers, SIMD instructions and even multi-threading... The Python code in programs using these libraries is not where the time is spent and the performance is comparable to that of compiled languages, while giving the programmer a simpler language to express their problems.
Question: Recently I read something about this and the point was that celery is more productive. Now, I can't find detailed information about the difference between these two and what should be the best way to use them.

Answer: Straight from the documentation:  If you need to perform heavy background computation and you don't necessarily need it to be run by the same process (for example, you don't need to share memory, variables, etc), you might benefit from using other bigger tools like Celery. They tend to require more complex configurations, a message/job queue manager, like RabbitMQ or Redis, but they allow you to run background tasks in multiple processes, and especially, in multiple servers. To see an example, check the Project Generators, they all include Celery already configured. But if you need to access variables and objects from the same FastAPI app, or you need to perform small background tasks (like sending an email notification), you can simply just use BackgroundTasks.  Have a look at this answer as well.
Question: I am trying to run Decision Tree Classifier but I face this problem.Please can you explain me how do I fix this Error?My English isn’t very good but I will try to understand!I'm just starting to learn the program, so please point me out if there's anything that isn't good enough.thank you! import matplotlib.pyplot as plt from sklearn.tree import DecisionTreeClassifier from sklearn import tree import pandas as pd sale=pd.read_csv('Online_Sale.csv') plt.rcParams['font.sans-serif'] = ['Microsoft JhengHei'] sale['回購'] = sale['回購'].apply(lambda x: 1 if x == 'Y' else 0) sale['單位售價'] = sale['單位售價'].str.replace(',', '').astype(float) x=sale['年紀'],sale['單位售價'] y=sale['回購'] print(x) print(y) clf = DecisionTreeClassifier(random_state=0) model = clf.fit(x, y) text_representation = tree.export_text(clf) print(text_representation) fig = plt.figure(figsize=(15,12)) tree.plot_tree(clf, filled=True) fig.savefig("decistion_tree.png")  data：  I looked for a lot of different approaches, but I didn't have a way to fully understand what the problem is...

Answer: There is just a small error with: x=sale['年紀'],sale['單位售價']  Rather than selecting the columns you want, this creates a tuple of the columns, hence the end of the error message ... does not match number of samples=2 One way to create a new pd.DataFrame with your selected columns: x=sale[['年紀', '單位售價']]
Question: So i want to calculate standard deviation excluding current group using groupby. Here an example of the data: import pandas as pd df = pd.DataFrame ({ 'group' : ['A','A','A','A','A','A','B','B','B','B','B','B'], 'team' : ['1','1','2','2','3','3','1','1','2','2','3','3',] 'value' : [1,2,5,7,2,3,7,8,8,9,6,4] })  For example, for group A team 1, i want to calculate the std dev of team 2 and 3, for group A team 2, i want to calculate the std dev of group 1 and 3, and so on. I managed to do it using groupby and apply but when using it on real data with literally milion of rows, it takes too long. So i am looking for a solution with vectorization. def std(row, data): data = data.loc[data['group']==row['group]] return data.groupby(['team']).filter(lambda x:(x['tool]!=row['team']).all())['value'].std() df['std_exclude'] = df.apply(lambda x: std(data=df),axis=1) 

Answer: You can use transform after combining group and team as a list: df['std'] = (df.assign(new=df[['group', 'team']].values.tolist())['new'].transform( lambda x: df[df['group'].eq(x[0]) & df['team'].ne(x[1])]['value'].std()))  Output: group team value std 0 A 1 1 2.217356 1 A 1 2 2.217356 2 A 2 5 0.816497 3 A 2 7 0.816497 4 A 3 2 2.753785 5 A 3 3 2.753785 6 B 1 7 2.217356 7 B 1 8 2.217356 8 B 2 8 1.707825 9 B 2 9 1.707825 10 B 3 6 0.816497 11 B 3 4 0.816497  There are some equal std values across different groups but you can verify that their std values are indeed equal.