eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I am trying to do a simple pivot on my dataframe, with one column as the index, one column as the columns, and one column as the values. Here is a screenshot of the code and the result:  As you can see, it is just one simple line of code. You can also notice that once the table is pivoted, the columns get placed into alphabetical order, which is not acceptable for the purposes of this program. This is supposed to be a general program for many use cases, so I can't reindex it by the column names like some other posts have suggested. Depending on the use case, it may have anywhere from 400 to 500 columns (later in the script these get put into smaller dataframes), but this one only has 32.  How can I fix this to keep the original sorting of the columns when I pivot without using the specific column names? I want to keep it general so it will work for most use cases. 

Answer: You can reorder the columns after the pivoting by selecting a column list in the original order. This column list is obtained from the original dataframe in generic way by selecting the column names for the first line.  Example: df = pd.DataFrame({'equipment name':['r1', 'r1', 'r2', 'r2'], 'name': ['col2', 'col1', 'col2', 'col1'], 'value': [1,2,3,4]}) #alphabetic order pd.pivot(df, 'equipment name', 'name', 'value') #name col1 col2 #equipment name #r1 2 1 #r2 4 3 #original order cols = df[df['equipment name']==df['equipment name'][0]].name.tolist() pd.pivot(df, 'equipment name', 'name', 'value')[cols] #name col2 col1 #equipment name #r1 1 2 #r2 3 4
Question: Is it possible to assign say infinity to something divided by 0 instead of it throwing ZeroDivisionError? Like a function that assigns infinity to something/0.

Answer: If you just want a float that represents infinity, you can issue float('inf') or float('-inf').  Standard Python floats and ints will give you a ZeroDivisionError, but you can use numpy datatypes. >>> import numpy as np >>> np.float64(15)/0 inf   Without numpy, write a function: def my_div(dividend, divisor): try: return dividend/divisor except ZeroDivisionError: if dividend == 0: raise ValueError('0/0 is undefined') # instead of raising an error, an alternative # is to return float('nan') as the result of 0/0 if dividend > 0: return float('inf') else: return float('-inf')
Question: I have a problem about file messages in python telegram bot. How can I receive file and read that file ? Or save it.

Answer: You can:  Register a handler that listens to Document get File object from the update (inside the listener using get_file) then simply call .download() to download the document  Here a sample code to get you started: from telegram.ext import Updater, MessageHandler, Filters BOT_TOKEN = ' ... ' def downloader(update, context): context.bot.get_file(update.message.document).download() # writing to a custom file with open("custom/file.doc", 'wb') as f: context.bot.get_file(update.message.document).download(out=f) updater = Updater(BOT_TOKEN, use_context=True) updater.dispatcher.add_handler(MessageHandler(Filters.document, downloader)) updater.start_polling() updater.idle()
Question: I was playing around with Unicode identifiers and stumbled upon this: >>> 𝑓, x = 1, 2 >>> 𝑓, x (1, 2) >>> 𝑓, f = 1, 2 >>> 𝑓, f (2, 2)  What's going on here? Why does Python replace the object referenced by 𝑓, but only sometimes? Where is that behavior described?

Answer: PEP 3131 -- Supporting Non-ASCII Identifiers says  All identifiers are converted into the normal form NFKC while parsing; comparison of identifiers is based on NFKC.  You can use unicodedata to test the conversions: import unicodedata unicodedata.normalize('NFKC', '𝑓') # f  which would indicate that '𝑓' gets converted to 'f' in parsing. Leading to the expected: 𝑓 = "Some String" print(f) # "Some String"
Question: Given the following data frame: pd.DataFrame({"A":[1,2,3],"B":[4,5,6],"C":[6,7,8]}) A B C 0 1 4 6 1 2 5 7 2 3 6 8 3 11 14 16 4 12 15 17 5 13 16 18  I would like to reshape it so it would look like so:  A B C A_1 B_1 C_1 A_2 B_2 C_2 0 1 4 6 2 5 7 3 6 8 1 11 14 16 12 15 17 13 16 18  So every 3 rows are grouped into 1 row How can I achieve this with pandas?

Answer: One idea is create MultiIndex with integer and modulo division and reshape by DataFrame.unstack: a = np.arange(len(df)) df.index = [a // 3, a % 3] df = df.unstack().sort_index(axis=1, level=1) df.columns = [f'{a}_{b}' for a, b in df.columns] print (df) A_0 B_0 C_0 A_1 B_1 C_1 A_2 B_2 C_2 0 1 4 6 2 5 7 3 6 8 1 11 14 16 12 15 17 13 16 18  For reverse operation is possible use str.split with DataFrame.stack: a = np.arange(len(df)) df1 = (df.set_index(pd.MultiIndex.from_arrays([a // 3, a % 3])) .unstack().sort_index(axis=1, level=1)) df1.columns = [f'{a}_{b}' for a, b in df1.columns] print (df1) A_0 B_0 C_0 A_1 B_1 C_1 A_2 B_2 C_2 0 1 4 6 2 5 7 3 6 8 1 11 14 16 12 15 17 13 16 18   df1.columns = df1.columns.str.split('_', expand=True) df2 = df1.stack().reset_index(drop=True) print (df2) A B C 0 1 4 6 1 2 5 7 2 3 6 8 3 11 14 16 4 12 15 17 5 13 16 18
Question: I essentially have a confusion matrix of size n x n with all my diagonal elements being 1.  For every row, I wish to calculate its mean, excluding the 1, i.e. excluding the diagonal value. Is there a simple way to do it in numpy? This is my current solution: mask = np.zeros(cs.shape, dtype=bool) np.fill_diagonal(mask, 1) print(np.ma.masked_array(cs, mask).mean(axis=1))  where cs is my n x n matrix The code seems convoluted, and I certainly feel that there's a much more elegant solution.

Answer: A concise one using summation - (cs.sum(1)-1)/(cs.shape[1]-1)  For a general case of ignoring diagonal elements, use np.diag in place of 1 offset - (cs.sum(1)-np.diag(cs))/(cs.shape[1]-1)  Another with mean - n = cs.shape[1] (cs.mean(1)-1./n)*(n/(n-1))
Question: I am using Python, Pandas for data analysis. I have sparsely distributed data in different columns like following | id | col1a | col1b | col2a | col2b | col3a | col3b | |----|-------|-------|-------|-------|-------|-------| | 1 | 11 | 12 | NaN | NaN | NaN | NaN | | 2 | NaN | NaN | 21 | 86 | NaN | NaN | | 3 | 22 | 87 | NaN | NaN | NaN | NaN | | 4 | NaN | NaN | NaN | NaN | 545 | 32 |  I want to combine this sparsely distributed data in different columns to tightly packed column like following. | id | group | cola | colb | |----|-------|-------|-------| | 1 | g1 | 11 | 12 | | 2 | g2 | 21 | 86 | | 3 | g1 | 22 | 87 | | 4 | g3 | 545 | 32 |  What I have tried is doing following, but not able to do it properly df['cola']=np.nan df['colb']=np.nan df['cola'].fillna(df.col1a,inplace=True) df['colb'].fillna(df.col1b,inplace=True) df['cola'].fillna(df.col2a,inplace=True) df['colb'].fillna(df.col2b,inplace=True) df['cola'].fillna(df.col3a,inplace=True) df['colb'].fillna(df.col3b,inplace=True)  But I think there must be more concise and efficient way way of doing this. How to do this in better way?

Answer: You can use df.stack() assuming 'id' is your index else set 'id' as index. Then use pd.pivot_table. df = df.stack().reset_index(name='val',level=1) df['group'] = 'g'+ df['level_1'].str.extract('col(\d+)') df['level_1'] = df['level_1'].str.replace('col(\d+)','') df.pivot_table(index=['id','group'],columns='level_1',values='val') level_1 cola colb id group 1 g1 11.0 12.0 2 g2 21.0 86.0 3 g1 22.0 87.0 4 g3 545.0 32.0
Question: Environment  Ubuntu 20.04 Python 3.7.3 Poetry 1.0.8  My Problem I installed poetry to manage packages, and I tried it with following simple project, . └── myproject ├── README.rst ├── myproject │ ├── __init__.py │ ├── main.py ├── myproject.egg-info │ ├── PKG-INFO │ ├── SOURCES.txt │ ├── dependency_links.txt │ ├── requires.txt │ └── top_level.txt ├── poetry.lock ├── pyproject.toml └── tests ├── __init__.py └── test_myproject.py  To run main.py I tried $ poetry run myproject/main.py  But I had an error, which says, [PermissionError] [Errno 13] Permission denied  What I tried To run my code, I tried another way. $ poetry shell (myproject-x8XipcUE-py3.7)$ python myproject/main.py  I had no error... What is the problem for my poetry run command?

Answer: My guess is that myproject/main.py isn't an executable (doesn't have the 'x') permission. That's why you can run it with python myproject/main.py, but can't run it as the main exe. To fix it, run chmod +x myproject/main.py, and then try poetry run again.  Of course, you'll have to have a proper Shebang at the very top of main.py. Something like #!/usr/bin/env python (again - at the very beginning of the file).
Question: I'm trying my hand at converting the following loop to a comprehension. Problem is given an input_list = [1, 2, 3, 4, 5] return a list with each element as multiple of all elements till that index starting from left to right. Hence return list would be [1, 2, 6, 24, 120]. The normal loop I have (and it's working): l2r = list() for i in range(lst_len): if i == 0: l2r.append(lst_num[i]) else: l2r.append(lst_num[i] * l2r[i-1]) 

Answer: Python 3.8+ solution:  := Assignment Expressions  lst = [1, 2, 3, 4, 5] curr = 1 out = [(curr:=curr*v) for v in lst] print(out)  Prints: [1, 2, 6, 24, 120]   Other solution (with itertools.accumulate): from itertools import accumulate out = [*accumulate(lst, lambda a, b: a*b)] print(out)
Question: I know there is the hovertemplate/hover_text/ option for traces (marker/line) but I cannot find such a thing for shapes. Is there a way to have a hover text pop up when moving over a shape? Maybe a workaround? Example: import plotly.graph_objects as go fig = go.Figure() fig.add_trace(go.Scatter( x=[1.5, 3], y=[2.5, 2.5], text=["Rectangle reference to the plot", "Rectangle reference to the axes"], mode="markers", )) fig.add_shape( # Rectangle reference to the plot type="rect", xref="paper", yref="paper", x0=0.25, y0=0, x1=0.5, y1=0.5, line=dict( color="LightSeaGreen", width=3, ), fillcolor="PaleTurquoise", )  When I hover over the two points, I get a hover-template with information. How can I get something similar for the shape?

Answer: I thought of a solution I am happy with. Simply draw a shape. You won't be able to see a hover text. However, if you add a trace with a fill on top of the shape, then set the trace to opacity=0 you will see the hover text from the trace pop up when moving over the shape. Again, thanks for your responses! import plotly.graph_objects as go # Draw shape (you won't be able to add a hover text for it) fig = go.Figure() fig.add_shape( type="rect", x0=0, y0=0, x1=4, y1=3, fillcolor='LightSkyBlue', line_color='Blue', name='Shape 1' ) # Adding a trace with a fill, setting opacity to 0 fig.add_trace( go.Scatter( x=[0,0,4,4,0], y=[0,3,3,0,0], fill="toself", mode='lines', name='', text='Custom text on top of shape', opacity=0 ) ) fig.show()
Question: I am working with tips data set, and here is the head of data set.  total_bill tip sex smoker day time size 0 16.99 1.01 Female No Sun Dinner 2 1 10.34 1.66 Male No Sun Dinner 3 2 21.01 3.50 Male No Sun Dinner 3 3 23.68 3.31 Male No Sun Dinner 2 4 24.59 3.61 Female No Sun Dinner 4  My code is sns.violinplot(x='day',y='total_bill',data=tips, hue=['sex','smoker'])  I want a violinplot of day with total_bill in which hue is sex and smoker, but I can not find any option to set multiple values of hue. Is there any way?

Answer: You could use a seaborn.catplot in order to use 'sex' as hue and 'smoker' as column for generating two side by side violinplot. Check this code: import seaborn as sns import matplotlib.pyplot as plt sns.set() tips = sns.load_dataset("tips") sns.catplot(x = "day", y = "total_bill", hue = "sex", col = "smoker", data = tips, kind = "violin", split = True) plt.show()  which gives me this plot:
Question: I am working on a Telegram Bot in Python but I struggle to use markdown correctly and I can not find any proper resources about the telegram markdown implementation. It gets even more complicated because of two different markdown "versions" (Markdown and Markdown_V2). And none of them is matching the behavior of the normal chat field (typing by hand). Test String: *Bold*, _italic_, *_bold and italic_*, **double bold**, __double italic__, __**double bold and double italic**__  parse_mode="Markdown": Bold, italic, _bold and italic_, double bold, double italic, double bold and double italic parse_mode="Markdown V2": Bold, italic, bold and italic, double bold, double italic, double bold and double italic in Chat: *Bold*, _italic_, *bold and italic*, double bold, double italic, **double bold and double italic** - How do I add bold and italic, and are there any other commands like underline and more? I need some explanation. Thanks.

Answer: Bots need a different markdown syntax. To send bold and italic text use: update.message.reply_text('*_bold and italic_*', parse_mode='MarkdownV2')  from the official telegram website https://core.telegram.org/bots/api#markdownv2-style *bold \*text* _italic \*text_ __underline__ ~strikethrough~ *bold _italic bold ~italic bold strikethrough~ __underline italic bold___ bold* [inline URL](http://www.example.com/) [inline mention of a user](tg://user?id=123456789) `inline fixed-width code` ``` pre-formatted fixed-width code block ``` ```python pre-formatted fixed-width code block written in the Python programming language ```  I recommend to use only MarkdownV2 syntax, since Markdown is less powerful
Question: I am trying to plot a facet_grid with stacked bar charts inside. I would like to use Seaborn. Its barplot function does not include a stacked argument. I tried to use FacetGrid.map with a custom callable function. import pandas as pd import seaborn as sns import numpy as np import matplotlib.pyplot as plt def custom_stacked_barplot(col_day, col_time, col_total_bill, **kwargs): dict_df={} dict_df['day']=col_day dict_df['time']=col_time dict_df['total_bill']=col_total_bill df_data_graph=pd.DataFrame(dict_df) df = pd.crosstab(index=df_data_graph['time'], columns=tips['day'], values=tips['total_bill'], aggfunc=sum) df.plot.bar(stacked=True) tips=sns.load_dataset("tips") g = sns.FacetGrid(tips, col='size', row='smoker') g = g.map(custom_stacked_barplot, "day", 'time', 'total_bill')  However I get an empty canvas and stacked bar charts separately. Empty canvas:  Graph1 apart:  Graph2:.  How can I fix this issue? Thanks for the help!

Answer: The simplest code to achive that result is this: import seaborn as sns import matplotlib.pyplot as plt sns.set() tips=sns.load_dataset("tips") g = sns.FacetGrid(tips, col = 'size', row = 'smoker', hue = 'day') g = (g.map(sns.barplot, 'time', 'total_bill', ci = None).add_legend()) plt.show()  which gives this result:
Question: Context I am updating an inherited repository which has poor test coverage. The repo itself is a pytest plugin. I've changed the repo to use tox along with pytest-cov, and converted the "raw" tests to use pytester as suggested in the pytest documentation when testing plugins. The testing and tox build, etc. works great. However, the coverage is reporting false misses with things like class definitions, imports, etc. This is because the code itself is being imported as part of pytest instantiation, and isn't getting "covered" until the testing actually starts. I've read pytest docs, pytest-cov and coverage docs, and tox docs, and tried several configurations, but to no avail. I've exhausted my pool of google keyword combinations that might lead me to a good solution. Repository layout pkg_root/ .tox/ py3/ lib/ python3.7/ site-pacakges/ plugin_module/ supporting_module.py plugin.py some_data.dat plugin_module/ supporting_module.py plugin.py some_data.dat tests/ conftest.py test_my_plugin.py tox.ini setup.py  Some relevant snippets with commentary: tox.ini [pytest] addopts = --cov={envsitepackagesdir}/plugin_module --cov-report=html testpaths = tests  This configuration gives me an error that no data was collected; no htmlcov is created in this case. If I just use --cov, I get (expected) very noisy coverage, which shows the functional hits and misses, but with the false misses reported above for imports, class definitions, etc. conftest.py pytest_plugins = ['pytester'] # Entire contents of file!  test_my_plugin.py def test_a_thing(testdir): testdir.makepyfile( """ def test_that_fixture(my_fixture): assert my_fixture.foo == 'bar' """ ) result = testdir.runpytest() result.assert_outcomes(passed=1)   How can I get an accurate report? Is there a way to defer the plugin loading until it's demanded by the pytester tests?

Answer: Instead of using the pytest-cov plugin, use coverage to run pytest: coverage run -m pytest ....  That way, coverage will be started before pytest.
Question: I want to install the requirements of this https://github.com/sraashis/deepdyn project, but when I run: pip install -r deepdyn/assets/requirements.txt  I receive the following error in the terminal: ERROR: Command errored out with exit status 1: command: /home/masoud/anaconda3/envs/tfgpu/bin/python -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-xzvdvhgj/MarkupSafe/setup.py'"'"'; __file__='"'"'/tmp/pip-install-xzvdvhgj/MarkupSafe/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-pip-egg-info-v7ebj7ab cwd: /tmp/pip-install-xzvdvhgj/MarkupSafe/ Complete output (5 lines): Traceback (most recent call last): File "<string>", line 1, in <module> File "/tmp/pip-install-xzvdvhgj/MarkupSafe/setup.py", line 6, in <module> from setuptools import setup, Extension, Feature ImportError: cannot import name 'Feature' from 'setuptools' (/home/masoud/anaconda3/envs/tfgpu/lib/python3.7/site-packages/setuptools/__init__.py) ---------------------------------------- ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output. 

Answer: The bug was fixed in version 1.1 but deepdyn requires version 1.0. This is probably a bug in deepdyn and should be reported. Or may be deepdyn requires some older version of setuptools. Again, ask the authors about it.
Question: The queue — A synchronized queue class simply states that   there are fewer functions allowed with SimpleQueue.  I need very basic queue functionality for a multithreading application, would it help in any way to use SimpleQueue?

Answer: queue.SimpleQueue handles more than threadsafe concurrency. It handles reentrancy - it is safe to call queue.SimpleQueue.put in precarious situations where it might be interrupting other work in the same thread. For example, you can safely call it from __del__ methods, weakref callbacks, or signal module signal handlers. If you need that, use queue.SimpleQueue.
Question: I am quite new to tensorflow, I would like to clearly know, what does the below command do? import tensorflow as tf from tensorflow import keras from tensorflow.keras import layers import os num_skipped = 0 for folder_name in ("Cat", "Dog"): print("folder_name:",folder_name) #folder_name: Cat folder_path = os.path.join("Dataset/PetImages", folder_name) print("folder_path:",folder_path) #folder_path: Dataset/PetImages/Cat for fname in os.listdir(folder_path): print("fname:",fname) #fname: 5961.jpg fpath = os.path.join(folder_path, fname) print("fpath:", fpath) #fpath: Dataset/PetImages/Cat/10591.jpg try: fobj = open(fpath, "rb") is_jfif = tf.compat.as_bytes("JFIF") in fobj.peek(10) finally: fobj.close() if not is_jfif: num_skipped += 1 # Delete corrupted image os.remove(fpath) print("Deleted %d images" % num_skipped)  Keras Website comment on the above code : When working with lots of real-world image data, corrupted images are a common occurence. Let's filter out badly-encoded images that do not feature the string "JFIF" in their header. I want to specifically know what does the below command do, how does it do ?  is_jfif = tf.compat.as_bytes("JFIF") in fobj.peek(10)  I checked the API but wasn't clearly able to understand it.  A better explanation will be of much help.  Thanks

Answer: Wikipedia explains that JPG files contain the string "JFIF" at the beginning of the file, encoded as bytes:  So:  tf.compat.as_bytes("JFIF") converts the string "JFIF" to bytes. You could also just use b"JFIF", though maybe the TensorFlow implementation has some optimization I don't know about. fobj.peek(10) theoretically returns the first 10 bytes of the file, but in practice it often returns the entire file. is_jfif then just checks if the converted "JFIF" string is in the result of fobj.peek.
Question: I want something of the form try: # code except *, error_message: print(error_message)  i.e I want to have a generic except block that catches all types of exceptions and prints an error message. Eg. "ZeroDivisionError: division by zero". Is it possible in python? If I do the following I can catch all exceptions, but I won't get the error message. try: # code except: print("Exception occurred") 

Answer: Try this:  except Exception as e: print(str(e))
Question: I am using DBSCAN for clustering. However, now I want to pick a point from each cluster that represents it, but I realized that DBSCAN does not have centroids as in kmeans. However, I observed that DBSCAN has something called core points. I am thinking if it is possible to use these core points or any other alternative to obtain a representative point from each cluster. I have mentioned below the code that I have used. import numpy as np from math import pi from sklearn.cluster import DBSCAN #points containing time value in minutes points = [100, 200, 600, 659, 700] def convert_to_radian(x): return((x / (24 * 60)) * 2 * pi) rad_function = np.vectorize(convert_to_radian) points_rad = rad_function(points) #generate distance matrix from each point dist = points_rad[None,:] - points_rad[:, None] #Assign shortest distances from each point dist[((dist > pi) & (dist <= (2*pi)))] = dist[((dist > pi) & (dist <= (2*pi)))] -(2*pi) dist[((dist > (-2*pi)) & (dist <= (-1*pi)))] = dist[((dist > (-2*pi)) & (dist <= (-1*pi)))] + (2*pi) dist = abs(dist) #check dist print(dist) #using default values, set metric to 'precomputed' db = DBSCAN(eps=((100 / (24*60)) * 2 * pi ), min_samples = 2, metric='precomputed') #check db print(db) db.fit(dist) #get labels labels = db.labels_ #get number of clusters no_clusters = len(set(labels)) - (1 if -1 in labels else 0) print('No of clusters:', no_clusters) print('Cluster 0 : ', np.nonzero(labels == 0)[0]) print('Cluster 1 : ', np.nonzero(labels == 1)[0]) print(db.core_sample_indices_)  I am happy to provide more details if needed.

Answer: Why don't you estimate the centroids of the resulted estimated clusters? points_of_cluster_0 = dist[labels==0,:] centroid_of_cluster_0 = np.mean(points_of_cluster_0, axis=0) print(centroid_of_cluster_0) points_of_cluster_1 = dist[labels==1,:] centroid_of_cluster_1 = np.mean(points_of_cluster_1, axis=0) print(centroid_of_cluster_1)