eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I have a data frame with one column that describes y-axis values and two more columns that describe the upper and lower bounds of a confidence interval. I would like to use those values to draw error bars using plotly. Now I am aware that plotly offers the possibility to draw confidence intervals (using the error_y and error_y_minus keyword-arguments) but not in the logic that I need, because those keywords are interpreted as additions and subtractions from the y-values. Instead, I would like to directly define the upper and lower positions: For instance, how could I use plotly and this example data frame import pandas as pd import plotly.express as px df = pd.DataFrame({'x':[0, 1, 2], 'y':[6, 10, 2], 'ci_upper':[8,11,2.5], 'ci_lower':[5,9,1.5]})  to produce a plot like this? 

Answer:  used Plotly Express to create bar chart used https://plotly.com/python/error-bars/#asymmetric-error-bars for generation of error bars using appropriate subtractions with your required outcome  import pandas as pd import plotly.express as px df = pd.DataFrame( {"x": [0, 1, 2], "y": [6, 10, 2], "ci_upper": [8, 11, 2.5], "ci_lower": [5, 9, 1.5]} ) px.bar(df, x="x", y="y").update_traces( error_y={ "type": "data", "symmetric": False, "array": df["ci_upper"] - df["y"], "arrayminus": df["y"] - df["ci_lower"], } )
Question: Can I use PyTorch tensors instead of NumPy arrays while working with scikit-learn? I tried some methods from scikit-learn like train_test_split and StandardScalar, and it seems to work just fine, but is there anything I should know when I'm using PyTorch tensors instead of NumPy arrays? According to this question on https://scikit-learn.org/stable/faq.html#how-can-i-load-my-own-datasets-into-a-format-usable-by-scikit-learn :  numpy arrays or scipy sparse matrices. Other types that are convertible to numeric arrays such as pandas DataFrame are also acceptable.  Does that mean using PyTorch tensors is completely safe?

Answer: I don't think PyTorch tensors are directly supported by scikit-learn. But you can always get the underlying numpy array from PyTorch tensors my_nparray = my_tensor.numpy()  and then use it with scikit learn functions.
Question: I've got a pandas dataframe that looks like this: mydict ={ 'person': ['Jenny', 'Jenny', 'David', 'David', 'Max', 'Max'], 'fruit': ['Apple', 'Orange', 'Apple', 'Orange', 'Apple', 'Orange'], 'eaten': [25, 75, 15, 5, 10, 10] } df = pd.DataFrame(mydict) person fruit eaten Jenny Apple 25 Jenny Orange 75 David Apple 15 David Orange 5 Max Apple 10 Max Orange 10  Which I'd like to convert into: person apple_percentage orange_percentage Jenny 0.25 0.75 David 0.75 0.25 Max 0.50 0.50  I'm guessing that I'll have to use groupby in some capacity to do this, but can't figure out a clean Pythonic way of doing so?

Answer: Use DataFrame.pivot with division by sums: df = df.pivot('person','fruit','eaten').add_suffix('_percentage') df = df.div(df.sum(axis=1), axis=0) print (df) fruit Apple_percentage Orange_percentage person David 0.75 0.25 Jenny 0.25 0.75 Max 0.50 0.50
Question: I've had a million and one issues with Poetry recently. I got it fully installed and working yesterday, but after a restart of my machine I'm back to having issues with it ;( Is there anyway to have Poetry consistently recognised in my Terminal, even after reboot?  System Specs:  Windows 10, Visual Studio Code, Bash - WSL Ubuntu CLI, Python 3.8.   Terminal: me@PF2DCSXD:/mnt/c/Users/me/Documents/GitHub/workers-python/workers/data_simulator/src$ poetry run python3 cli.py poetry: command not found me@PF2DCSXD:/mnt/c/Users/me/Documents/GitHub/workers-python/workers/data_simulator/src$ curl -sSL https://raw.githubusercontent.com/python-poetry/poetry/master/get-poetry.py | python3 Retrieving Poetry metadata This installer is deprecated. Poetry versions installed using this script will not be able to use 'self update' command to upgrade to 1.2.0a1 or later. Latest version already installed. me@PF2DCSXD:/mnt/c/Users/me/Documents/GitHub/workers-python/workers/data_simulator/src$ poetry run python3 cli.py poetry: command not found me@PF2DCSXD:/mnt/c/Users/me/Documents/GitHub/workers-python/workers/data_simulator/src$  Please let me know if there is anything else I can add to post to help further clarify.

Answer: When I run this, after shutdown of bash Terminal: export PATH="$HOME/.poetry/bin:$PATH"  poetry command is then recognised. However, this isn't enough alone; as every time I shutdown the terminal I need to run the export. Possibly needs to be saved in a file.
Question: This question is to clarify my doubts related to python typing from typing import Union class ParentClass: parent_prop = 1 class ChildA(ParentClass): child_a_prop = 2 class ChildB(ParentClass): child_b_prop = 3 def method_body(val) -> ParentClass: if val: return ChildA() else: return ChildB() def another_method() -> ChildA: return method_body(True) print(another_method().child_a_prop)  In the above piece of code, the linting tool I used is printing error as below error: Incompatible return value type (got "ParentClass", expected "ChildA")  (where I do method_body(True)) I have also set the method_body return type as  Union[ChildA, ChildB]. This will result error: Incompatible return value type (got "Union[ChildA, ChildB]", expected "ChildA") I am looking for a better way to do this. If anyone knows the solution, your help will be very much appreciated.

Answer: mypy does not do runtime analysis so it cannot guess that calling method_body with argument True will always result in a ChildA object. So the error it produces does make sense. You have to guide mypy in some way to tell him that you know what you are doing and that another_method indeed produces a ChildA object when called with argument True. One is to use a cast: from typing import cast def another_method() -> ChildA: return cast(ChildA, method_body(True))  Another one is to add an assertion: def another_method() -> ChildA: result = method_body(True) assert isinstance(result, ChildA) return result  The difference between the two is that the cast does not have any runtime implication. You can think of it as a comment put here to guide mypy in its checks, but the cast function only returns its second parameter, ie, here is the body of cast: def cast(typ, val): return val  Whereas the assert can naturally raise an AssertionError error (not in that case obviously, but in general).
Question: I am bit confused between both the methods : concatenate and stack The concatenate and stack provides exactly same output , what is the difference between both of them ? Using : concatenate import numpy as np my_arr_1 = np.array([ [1,4] , [2,7] ]) my_arr_2 = np.array([ [0,5] , [3,8] ]) join_array=np.concatenate((my_arr_1,my_arr_2),axis=0) print(join_array)  Using : stack import numpy as np my_arr_1 = np.array([ [1,4] , [2,7] ]) my_arr_2 = np.array([ [0,5] , [3,8] ]) join1_array=np.stack((my_arr_1,my_arr_2),axis=0) print(join1_array)  Output for both is same : [[[1 4] [2 7]] [[0 5] [3 8]]] 

Answer: In [160]: my_arr_1 = np.array([ [1,4] , [2,7] ]) ...: my_arr_2 = np.array([ [0,5] , [3,8] ]) ...: ...: join_array=np.concatenate((my_arr_1,my_arr_2),axis=0) In [161]: join_array Out[161]: array([[1, 4], [2, 7], [0, 5], [3, 8]]) In [162]: _.shape Out[162]: (4, 2)  concatenate joined the 2 arrays on an existing axis, so the (2,2) become (4,2). In [163]: join1_array=np.stack((my_arr_1,my_arr_2),axis=0) In [164]: join1_array Out[164]: array([[[1, 4], [2, 7]], [[0, 5], [3, 8]]]) In [165]: _.shape Out[165]: (2, 2, 2)  stack joined them on a new axis. It actually made them both (1,2,2) shape, and then used concatenate. The respective docs should make this clear.
Question: @pytest.fixture def d_service(): c = DService() return c # @pytest.mark.asyncio # tried it too async def test_get_file_list(d_service): files = await d_service.get_file_list('') print(files)  However, it got the following error?  collected 0 items / 1 errors =================================== ERRORS ==================================== ________________ ERROR collecting tests/e2e_tests/test_d.py _________________ ..\..\..\..\..\anaconda3\lib\site-packages\pluggy\__init__.py:617: in __call__ return self._hookexec(self, self._nonwrappers + self._wrappers, kwargs) ..\..\..\..\..\anaconda3\lib\site-packages\pluggy\__init__.py:222: in _hookexec return self._inner_hookexec(hook, methods, kwargs) ..\..\..\..\..\anaconda3\lib\site-packages\pluggy\__init__.py:216: in firstresult=hook.spec_opts.get('firstresult'), ..\..\..\..\..\anaconda3\lib\site-packages\_pytest\python.py:171: in pytest_pycollect_makeitem res = outcome.get_result() ..\..\..\..\..\anaconda3\lib\site-packages\anyio\pytest_plugin.py:98: in pytest_pycollect_makeitem marker = collector.get_closest_marker('anyio') E AttributeError: 'Module' object has no attribute 'get_closest_marker' !!!!!!!!!!!!!!!!!!! Interrupted: 1 errors during collection !!!!!!!!!!!!!!!!!!! =========================== 1 error in 2.53 seconds ===========================   I installed the following package. The error is gone but the test is skipped. pip install pytest-asyncio   (base) PS>pytest -s tests\e2e_tests\test_d.py ================================================================================================================== test session starts =================================================================================================================== platform win32 -- Python 3.6.4, pytest-6.2.5, py-1.11.0, pluggy-1.0.0 rootdir: C:\Users\X01324908\source\rds\research_data_science\sftp\file_handler plugins: anyio-3.3.4, asyncio-0.16.0 collected 1 item tests\e2e_tests\test_d.py s ==================================================================================================================== warnings summary ==================================================================================================================== tests/e2e_tests/test_d.py::test_get_file_list c:\users\x01324908\anaconda3\lib\site-packages\_pytest\python.py:172: PytestUnhandledCoroutineWarning: async def functions are not natively supported and have been skipped. You need to install a suitable plugin for your async framework, for example: - anyio - pytest-asyncio - pytest-tornasync - pytest-trio - pytest-twisted warnings.warn(PytestUnhandledCoroutineWarning(msg.format(nodeid))) -- Docs: https://docs.pytest.org/en/stable/warnings.html ============= 

Answer: This works for me, please try: import asyncio import pytest pytest_plugins = ('pytest_asyncio',) @pytest.mark.asyncio async def test_simple(): await asyncio.sleep(0.5)  Output of pytest -v confirms it passes: collected 1 item test_async.py::test_simple PASSED  And I have installed: pytest 6.2.5 pytest-asyncio 0.16.0 # anyio not installed
Question: I have a dataframe like below:     Feature value frequency label     age_45_and_above No 2700 negative   age_45_and_above No 1707 positive   age_45_and_above No 83 other   age_45_and_above Yes 222 negative   age_45_and_above Yes 15 positive   age_45_and_above Yes 8 other   age_45_and_above [Null] 323 negative   age_45_and_above [Null] 8 other   age_45_and_above [Null] 5 positive   talk No 20 negative   talk No 170 positive   talk No 500 other   talk Yes 210 negative   talk Yes 1500 positive   talk Yes 809 other   talk [Null] 234 negative   talk [Null] 43 other   talk [Null] 85 positive     and so on. for each feature group, I want to find the maximum frequency with all its related row data, like if the feature is age_45_and_above then by looking for NO group we have 3 rows with different frequency and label, I want to report the maximum one with it's related data. I've tried groupby in different ways: result.groupby(['Feature', 'Value'])['Frequency', 'Predict'].max()  or this one, with this one, I'm getting multi-Index dataframe which is not the desired results: result.groupby(['Feature', 'Value', 'Predict'])['Frequency'].max()  and so many failed attempts with idxmax, transfrom and ... . the intended output I'm looking for looks like this:     Feature value frequency label     age_45_and_above No 2700 negative   age_45_and_above Yes 222 negative   age_45_and_above [Null] 323 negative   talk No 500 other   talk Yes 1500 positive   talk [Null] 234 negative     Also, I wonder how to sum the frequencies for each <<Feature-value>> group except the max row as I don't know how to locate the max row, like in here for the first feature and value, <<age_45_and_above-No>> max is 2700, so the sum would be 1707+83. Thanks for your time.

Answer: I would do it by using merge on the grouped data. Based on this data: df = pd.DataFrame({'Feature':['age']*9+['talk']*9, 'value':(['No']*3+['Yes']*3+['[Null]']*3)*2, 'frequency':[2700,1707,83,222,15,8,323,8,5,20,170,500,210,1500,809,234,43,85], 'label':['N','P','O']*6})  Using: df.groupby(['Feature','value'],as_index=False)['frequency'].max().merge(df,on=['Feature','Value','frequency'])  Outputs:  Feature value frequency label 0 age No 2700 N 1 age Yes 222 N 2 age [Null] 323 N 3 talk No 500 O 4 talk Yes 1500 P 5 talk [Null] 234 N  Adding the extra column can be done via a simple assignment: df_1['sum_no_max'] = df.groupby(['Feature','value'])['frequency'].sum().values - df_1['frequency'].values  Finally outputting:  Feature value frequency label sum_no_max 0 age No 2700 N 1790 1 age Yes 222 N 23 2 age [Null] 323 N 13 3 talk No 500 O 190 4 talk Yes 1500 P 1019 5 talk [Null] 234 N 128
Question: I have to fill a column only if all the values of that column are null. For example c df = pd.DataFrame(data = {"col1":[3, np.nan, np.nan, 21, np.nan], "col2":[4, np.nan, 12, np.nan, np.nan], "col3":[33, np.nan, 55, np.nan, np.nan], "col4":[np.nan, np.nan, np.nan, np.nan, np.nan]}) >>> df col1 col2 col3 col4 0 3.0 4.0 33.0 NaN 1 NaN NaN NaN NaN 2 NaN 12.0 55.0 NaN 3 21.0 NaN NaN NaN 4 NaN NaN NaN NaN  In the above example, I have to replace the values of col4 with 100 since all the values are null/NaN. So for the above example. I have to get the output as below.  col1 col2 col3 col4 0 3.0 4.0 33.0 100 1 NaN NaN NaN 100 2 NaN 12.0 55.0 100 3 21.0 NaN NaN 100 4 NaN NaN NaN 100  Tried using the below command. But its replacing values of a column only if it contains atleast 1 non-nan value df.where(df.isnull().all(axis=1), df.fillna(100), inplace=True) Could you please let me know how to do this. Thanks

Answer: Use indexing: df.loc[:, df.isna().all()] = 100 print(df) # Output: col1 col2 col3 col4 0 3.0 4.0 33.0 100.0 1 NaN NaN NaN 100.0 2 NaN 12.0 55.0 100.0 3 21.0 NaN NaN 100.0 4 NaN NaN NaN 100.0
Question: Consider: def my_max(*a): n = len(a) max_v = a[0] for i in range (1,n): if a[i] > max_v: max_v = a[i] return max_v def my_min(*a): n = len(a) min_v = a[0] for i in range (1,n): if a[i] < min_v: min_v = a[i] return min_v test = [7, 4, 2, 6, 8] assert max(test) == my_max(test) and min(test) == my_min(test) assert max(7, 4, 2, 5) == my_max(7, 4, 2, 5) and min(7, 4, 2, 5) == my_min(7, 4, 2, 5) print("pass")  I am trying to write the max() function of Python in code. If I add the asterisk in front of the input, it won't pass the first assertion. If I don't it wouldn't pass the second assertion. What should I write in the input for it to pass both assertions, like it does in the max() function of Python?

Answer: Short answer: Use a star to collect the arguments in a tuple and then add a special case for a tuple of length one to handle a single iterable argument. Source material: The C code that handles the logic can be found at: https://github.com/python/cpython/blob/da20d7401de97b425897d3069f71f77b039eb16f/Python/bltinmodule.c#L1708 Simplified pure python code: If you ignore the default and key keyword arguments, what's left simplifies to: def mymax(*args): if len(args) == 0: raise TypeError('max expected at least 1 argument, got 0') if len(args) == 1: args = tuple(args[0]) largest = args[0] for x in args[1:]: if x > largest: largest = x return largest  There are other nuances, but this should get you started. Documentation: The special handling for the length one case versus other cases is documented here:  Return the largest item in an iterable or the largest of two or more arguments. If one positional argument is provided, it should be an iterable. The largest item in the iterable is returned. If two or more positional arguments are provided, the largest of the positional arguments is returned.  More complete version: This includes some of aforementioned nuances like the key and default keyword arguments and the use of iterators instead of slices: sentinel = object() def mymax(*args, default=sentinel, key=None): """max(iterable, *[, default=obj, key=func]) -> value max(arg1, arg2, *args, *[, key=func]) -> value With a single iterable argument, return its biggest item. The default keyword-only argument specifies an object to return if the provided iterable is empty. With two or more arguments, return the largest argument. """ if not args: raise TypeError('max expected at least 1 argument, got 0') if len(args) == 1: it = iter(args[0]) else: if default is not sentinel: raise TypeError('Cannot specify a default for max() with multiple positional arguments') it = iter(args) largest = next(it, sentinel) if largest is sentinel: if default is not sentinel: return default raise ValueError('max() arg is an empty sequence') if key is None: for x in it: if x > largest: largest = x return largest largest_key = key(largest) for x in it: kx = key(x) if kx > largest_key: largest = x largest_key = kx return largest # This makes the tooltips nicer # but isn't how the C code actually works # and it is only half correct. mymax.__text_signature__ = '($iterable, /, *, default=obj, key=func)'
Question: I was wanting to upgrade my Python version (to 3.10 in this case), so after installing Python 3.10, I proceeded to try adding some modules I use, e.g., opencv, which ran into: python3.10 -m pip install opencv-python  Output: Traceback (most recent call last): File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main return _run_code(code, main_globals, None, File "/usr/lib/python3.10/runpy.py", line 86, in _run_code exec(code, run_globals) File "/usr/lib/python3/dist-packages/pip/__main__.py", line 16, in <module> from pip._internal.cli.main import main as _main # isort:skip # noqa File "/usr/lib/python3/dist-packages/pip/_internal/cli/main.py", line 10, in <module> from pip._internal.cli.autocompletion import autocomplete File "/usr/lib/python3/dist-packages/pip/_internal/cli/autocompletion.py", line 9, in <module> from pip._internal.cli.main_parser import create_main_parser File "/usr/lib/python3/dist-packages/pip/_internal/cli/main_parser.py", line 7, in <module> from pip._internal.cli import cmdoptions File "/usr/lib/python3/dist-packages/pip/_internal/cli/cmdoptions.py", line 19, in <module> from distutils.util import strtobool ModuleNotFoundError: No module named 'distutils.util'  And sudo apt-get install python3-distutils  Output: [sudo] password for jeremy: Reading package lists... Done Building dependency tree Reading state information... Done python3-distutils is already the newest version (3.8.10-0ubuntu1~20.04). ... 0 upgraded, 0 newly installed, 0 to remove and 4 not upgraded.  Since distutils already seems to be installed, I can't grok how to proceed.

Answer: It looks like distutils has versioning, so after the following, it seems to be able to proceed. sudo apt-get install python3.10-distutils  Output: Reading package lists... Done Building dependency tree Reading state information... Done ... Setting up python3.10-lib2to3 (3.10.0-1+focal1) ... Setting up python3.10-distutils (3.10.0-1+focal1) ...  And: python3.10 -m pip install opencv-python
Question: I would like to be able to run an ad-hoc python script that would access and run analytics on the model calculated by a dbt run, are there any best practices around this?

Answer: We recently built a tool that could that caters very much to this scenario. It leverages the ease of referencing tables from dbt in Python-land. It's called dbt-fal. The idea is that you would define the python scripts you would like to run after your dbt models are run: # schema.yml models: - name: iris meta: owner: "@matteo" fal: scripts: - "notify.py"  And then the file notify.py is called if the iris model was run in the last dbt run: # notify.py import os from slack_sdk import WebClient from slack_sdk.errors import SlackApiError CHANNEL_ID = os.getenv("SLACK_BOT_CHANNEL") SLACK_TOKEN = os.getenv("SLACK_BOT_TOKEN") client = WebClient(token=SLACK_TOKEN) message_text = f"""Model: {context.current_model.name} Status: {context.current_model.status} Owner: {context.current_model.meta['owner']}""" try: response = client.chat_postMessage( channel=CHANNEL_ID, text=message_text ) except SlackApiError as e: assert e.response["error"]  Each script is ran with a reference to the current model for which it is running in a context variable.  To start using fal, just pip install fal and start writing your python scripts.
Question: I have the following data set: data = [6.92, 1.78, 0.0, 0.0, 3.5, 8.82, 3.06, 0.0, 0.0, 5.54, -10.8, -6.03, 0.0, 0.0, -6.8, 13.69, 8.61, 9.98, 0.0, 9.42, 4.91, 3.54, 2.62, 5.65, 1.95, 8.91, 11.46, 5.31, 6.93, 6.42]  Is there a way to remove the 0.0 labels from the bar plot?  I tried df = df.replace(0, "") but then I get a list index out of range error code. My code: import pandas as pd import matplotlib.pyplot as plt import numpy as np data = [6.92, 1.78, 0.0, 0.0, 3.5, 8.82, 3.06, 0.0, 0.0, 5.54, -10.8, -6.03, 0.0, 0.0, -6.8, 13.69, 8.61, 9.98, 0.0, 9.42, 4.91, 3.54, 2.62, 5.65, 1.95, 8.91, 11.46, 5.31, 6.93, 6.42] df = pd.DataFrame(np.array(data).reshape(6,5), columns=['Bank1', 'Bank2', 'Bank3', 'Bank4', 'Bank5'], index =['2016', '2017', '2018', '2019', '2020', '2021']) print(df) ax = df.plot(kind='bar', rot=0, xlabel='Year', ylabel='Total Return %', title='Overall Performance', figsize=(15, 10)) ax.bar_label(ax.containers[0], fmt='%.1f', fontsize=8, padding=3) ax.bar_label(ax.containers[1], fmt='%.1f', fontsize=8, padding=3) ax.bar_label(ax.containers[2], fmt='%.1f', fontsize=8, padding=3) ax.bar_label(ax.containers[3], fmt='%.1f', fontsize=8, padding=3) ax.bar_label(ax.containers[4], fmt='%.1f', fontsize=8, padding=3) ax.legend(title='Columns', bbox_to_anchor=(1, 1.02), loc='upper left') plt.show() 

Answer:  labels passed to matplotlib.pyplot.bar_label must be customized  Adjust the comparison (!= 0) value or range as needed.   labels = [f'{v.get_height():0.0f}' if v.get_height() != 0 else '' for v in c ] without the assignment expression (:=). See this answer for additional details and examples using .bar_label. Tested in python v3.12.0, pandas v2.1.2, matplotlib v3.8.1.  import pandas as pd import matplotlib.pyplot as plt data = [6.92, 1.78, 0.0, 0.0, 3.5, 8.82, 3.06, 0.0, 0.0, 5.54, -10.8, -6.03, 0.0, 0.0, -6.8, 13.69, 8.61, 9.98, 0.0, 9.42, 4.91, 3.54, 2.62, 5.65, 1.95, 8.91, 11.46, 5.31, 6.93, 6.42] df = pd.DataFrame(np.array(data).reshape(6,5), columns=['Bank1', 'Bank2', 'Bank3', 'Bank4', 'Bank5'], index =['2016', '2017', '2018', '2019', '2020', '2021']) ax = df.plot(kind='bar', rot=0, xlabel='Year', ylabel='Total Return %', title='Overall Performance', width=0.9, figsize=(15, 10)) for c in ax.containers: # customize the label to account for cases when there might not be a bar section labels = [f'{h:0.1f}' if (h := v.get_height()) != 0 else '' for v in c ] # set the bar label ax.bar_label(c, labels=labels, fontsize=8, padding=3) # from matplotlib 3.7, the following options with fmt also work # ax.bar_label(c, fmt=lambda x: np.where(x != 0, x, ''), fontsize=8, padding=3) # ax.bar_label(c, fmt=lambda x: f'{x:0.2f}' if x != 0 else '', fontsize=8, padding=3) ax.legend(title='Banks', bbox_to_anchor=(1, 0.5), loc='center left', frameon=False) plt.show()