eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: Below is my workflow file, which worked previously forever and I've not changed anything. env: PYTHON_VERSION: '3.8.9' jobs: build: name: build 🔧 runs-on: ubuntu-latest steps: - uses: actions/checkout@v3 - uses: actions/setup-python@v3 with: python-version: ${{ env.PYTHON_VERSION }} - run: mkdir file-icons product-icons - run: python3 -m translate - uses: actions/upload-artifact@v3 with: name: build path: | file-icons/ product-icons/ thumbnails/ *.md package.json  today it decided to stop working 🥲 Error: Version 3.8.9 with arch x64 not found  this is with debug logging on ##[debug]x64===x64 && linux===linux makes no sense to me, it found a match and then just moved on  what it was before 

Answer: Resolved. GitHub actions recently updated ubuntu-latest from 20.04 to 22.04  https://github.blog/changelog/2022-08-09-github-actions-ubuntu-22-04-is-now-generally-available-on-github-hosted-runners/ https://github.com/actions/runner-images/issues/6399  certain versions of python is not yet supported by setup-python action on 22.04 (and currently they are not looking to support old python versions such as 3.8.9) Solutions: option one - set ubuntu to 20.0.4 (the suggested option according to the runner image issue on GitHub https://github.com/actions/runner-images/issues/6399) option two - use python 3.8.12 or newer
Question: Goal: create a single-dispatch generic function; as per functools documentation. I want to use my_func() to calculate dtypes: int or list, in pairs. Note: I've chosen to implement type hints and to raise errors for my own test cases, outside of the scope of this post. For many argument data type hints; this post uses many @my_func.register(...) decorators. Code: from functools import singledispatch from typeguard import typechecked from typing import Union @singledispatch @typechecked def my_func(x, y): raise NotImplementedError(f'{type(x)} and or {type(y)} are not supported.') @my_func.register(int) def my_func(x: int, y: int) -> Union[int, str]: try: return round(100 * x / (x + y)) except (ZeroDivisionError, TypeError, AssertionError) as e: return f'{e}' @my_func.register(list) def my_func(x: list, y: list) -> Union[int, str]: try: return round(100 * sum(x) / (sum(x) + sum(y))) except (ZeroDivisionError, TypeError, AssertionError) as e: return f'{e}' a = my_func(1, 2) print(a) b = my_func([0, 1], [2, 3]) print(b)  (venv) me@ubuntu-pcs:~/PycharmProjects/project$ python3 foo/bar.py /home/me/miniconda3/envs/venv/lib/python3.9/site-packages/typeguard/__init__.py:1016: UserWarning: no type annotations present -- not typechecking __main__.my_func warn('no type annotations present -- not typechecking {}'.format(function_name(func))) Traceback (most recent call last): File "/home/me/PycharmProjects/project/foo/bar.py", line 22, in <module> @my_func.register(list) AttributeError: 'function' object has no attribute 'register' 

Answer: Functions must be uniquely named Credit to @Bijay Regmi for pointing this out. @typechecked is placed above only the polymorphed @my_func.register functions; not above the @singledispatch function. Note: you still invoke my_func(); I am just testing the polymorphed functions. from functools import singledispatch from typeguard import typechecked from typing import Union @singledispatch def my_func(x, y): raise NotImplementedError(f'{type(x)} and or {type(y)} are not supported.') @my_func.register(int) @typechecked def my_func_int(x: int, y: int) -> Union[int, str]: try: return round(100 * x / (x + y)) except (ZeroDivisionError, TypeError, AssertionError) as e: return f'{e}' @my_func.register(list) @typechecked def my_func_list(x: list, y: list) -> Union[int, str]: try: return round(100 * sum(x) / (sum(x) + sum(y))) except (ZeroDivisionError, TypeError, AssertionError) as e: return f'{e}' a = my_func_int(1, 2) print(a) b = my_func_list([0, 1], [2, 3]) print(b)  (venv) me@ubuntu-pcs:~/PycharmProjects/project$ python3 foo/bar.py 33 17
Question: I have my DevOps pipeline which is using pytest to execute unit tests found in Python code. I'm using a root folder called "core" for the main python functionality, and reference it using the following format: import unittest from core.objects.paragraph import Paragraph from core.objects.sentence import Sentence  Running this locally (python -m pytest) works correctly and executes all of the tests. However, everytime I run this from Azure DevOps I get the error message: ModuleNotFoundError: No module named 'core'  I have no idea why this is happening 🤷‍♂️🤷‍♂️🤷‍♂️ For reference, this is my YAML build definition trigger: - main pool: vmImage: ubuntu-latest strategy: matrix: Python310: python.version: '3.10.7' steps: - task: UsePythonVersion@0 inputs: versionSpec: '$(python.version)' displayName: 'Use Python $(python.version)' - script: | python -m pip install --upgrade pip pip install -r requirements.txt displayName: 'Install dependencies' - script: | pip install pytest pytest-azurepipelines displayName: 'Install pytest' - script: | python -m pytest --junitxml=$(Build.StagingDirectory)/Testoutput.xml workingDirectory: 'tests' displayName: 'Test with PyTest'  Once the execution has finished, this is the error message I'm seeing: ____________ ERROR collecting test_objects/test_paragraph_object.py ____________ ImportError while importing test module '/home/vsts/work/1/s/tests/test_objects/test_paragraph_object.py'. Hint: make sure your test modules/packages have valid Python names. Traceback: /opt/hostedtoolcache/Python/3.10.7/x64/lib/python3.10/importlib/__init__.py:126: in import_module return _bootstrap._gcd_import(name[level:], package, level) test_objects/test_paragraph_object.py:2: in <module> from core.objects.paragraph import Paragraph E ModuleNotFoundError: No module named 'core' ------------ generated xml file: /home/vsts/work/1/a/Testoutput.xml ------------ 

Answer: Please make sure the directory/home/vsts/work/1/s/tests/has a __init__.pyfile.
Question: So I want to show this data in just two columns. For example, I want to turn this data  Year Jan Feb Mar Apr May Jun 1997 3.45 2.15 1.89 2.03 2.25 2.20 1998 2.09 2.23 2.24 2.43 2.14 2.17 1999 1.85 1.77 1.79 2.15 2.26 2.30 2000 2.42 2.66 2.79 3.04 3.59 4.29  into this  Date Price Jan-1977 3.45 Feb-1977 2.15 Mar-1977 1.89 Apr-1977 2.03 .... Jan-2000 2.42 Feb-2000 2.66  So far, I have read about how to combine two columns into another dataframe using .apply() .agg(), but no info how to combine them as I showed above. import pandas as pd df = pd.read_csv('matrix-A.csv', index_col =0 ) matrix_b = ({}) new = pd.DataFrame(matrix_b) new["Date"] = df['Year'].astype(float) + "-" + df["Dec"] print(new)  I have tried this way, but it of course does not work. I have also tried using pd.Series() but no success I want to ask whether there is any site where I can learn how to do this, or does anybody know correct way to solve this?

Answer: Another possible solution, which is based on pandas.DataFrame.stack: out = df.set_index('Year').stack() out.index = ['{}_{}'.format(j, i) for i, j in out.index] out = out.reset_index() out.columns = ['Date', 'Value']  Output:  Date Value 0 Jan_1997 3.45 1 Feb_1997 2.15 2 Mar_1997 1.89 3 Apr_1997 2.03 4 May_1997 2.25 .... 19 Feb_2000 2.66 20 Mar_2000 2.79 21 Apr_2000 3.04 22 May_2000 3.59 23 Jun_2000 4.29
Question: I have a asyncio running loop, and from the coroutine I'm calling a sync function, is there any way we can call and get result from an async function in a sync function tried below code, it is not working want to print output of hel() in i() without changing i() to async function is it possible, if yes how? import asyncio async def hel(): return 4 def i(): loop = asyncio.get_running_loop() x = asyncio.run_coroutine_threadsafe(hel(), loop) ## need to change y = x.result() ## this lines print(y) async def h(): i() asyncio.run(h()) 

Answer: This is one of the most commonly asked type of question here. The tools to do this are in the standard library and require only a few lines of setup code. However, the result is not 100% robust and needs to be used with care. This is probably why it's not already a high-level function. The basic problem with running an async function from a sync function is that async functions contain await expressions. Await expressions pause the execution of the current task and allow the event loop to run other tasks. Therefore async functions (coroutines) have special properties that allow them to yield control and resume again where they left off. Sync functions cannot do this. So when your sync function calls an async function and that function encounters an await expression, what is supposed to happen? The sync function has no ability to yield and resume. A simple solution is to run the async function in another thread, with its own event loop. The calling thread blocks until the result is available. The async function behaves like a normal function, returning a value. The downside is that the async function now runs in another thread, which can cause all the well-known problems that come with threaded programming. For many cases this may not be an issue. This can be set up as follows. This is a complete script that can be imported anywhere in an application. The test code that runs in the if __name__ == "__main__" block is almost the same as the code in the original question. The thread is lazily initialized so it doesn't get created until it's used. It's a daemon thread so it will not keep your program from exiting. The solution doesn't care if there is a running event loop in the main thread. import asyncio import threading _loop = asyncio.new_event_loop() _thr = threading.Thread(target=_loop.run_forever, name="Async Runner", daemon=True) # This will block the calling thread until the coroutine is finished. # Any exception that occurs in the coroutine is raised in the caller def run_async(coro): # coro is a couroutine, see example if not _thr.is_alive(): _thr.start() future = asyncio.run_coroutine_threadsafe(coro, _loop) return future.result() if __name__ == "__main__": async def hel(): await asyncio.sleep(0.1) print("Running in thread", threading.current_thread()) return 4 def i(): y = run_async(hel()) print("Answer", y, threading.current_thread()) async def h(): i() asyncio.run(h())  Output: Running in thread <Thread(Async Runner, started daemon 28816)> Answer 4 <_MainThread(MainThread, started 22100)>
Question: I am trying to build a tool that will essentially scramble a dataset while maintaining the same elements. For example, if I have the table below  1 2 3 4 5 6 0 ABC 1234 NL00 Paid VISA 1 BCD 2345 NL01 Unpaid AMEX 2 CDE 3456 NL02 Unpaid VISA  I want it to then look go through each column, pick a random value, and paste that into a new df. An example output would be  1 2 3 4 5 6 2 BCD 2345 NL01 Unpaid VISA 0 BCD 1234 NL02 Unpaid VISA 0 CDE 3456 NL01 Paid VISA  I have managed to make it work with the code below, although for 24 columns the code was quite repetitive and I know a loop should be able to do this much quicker, I just have not been able to make it work. import pandas as pd import random lst1 = df['1'].to_list() lst2 = df['2'].to_list() lst3 = df['3'].to_list() lst4 = df['4'].to_list() lst5 = df['5'].to_list() lst6 = df['6'].to_list() df_new = pd.DataFrame() df_new['1'] = random.choices(lst1, k=2000) df_new['2'] = random.choices(lst2, k=2000) df_new['3'] = random.choices(lst3, k=2000) df_new['4'] = random.choices(lst4, k=2000) df_new['5'] = random.choices(lst5, k=2000) df_new['6'] = random.choices(lst6, k=2000) 

Answer: Here's an easy solution: df.apply(pd.Series.sample, replace=True, ignore_index=True, frac=1)  Output (potential):  1 2 3 4 5 6 0 2 CDE 3456 NL00 Paid VISA 1 2 BCD 3456 NL01 Paid VISA 2 0 CDE 3456 NL01 Paid VISA  pd.DataFrame.apply applies pd.Series.sample method to each column of the dataframe with resampling (replace=True) and return 100% size of the original dataframe with frac=1.
Question: I need to delete duplicated rows based on combination of two columns (person1 and person2 columns) which have strings. For example person1: ryan and person2: delta or person 1: delta and person2: ryan is same and provides the same value in messages column. Need to drop one of these two rows. Return the non duplicated rows as well. Code to recreate df df = pd.DataFrame({"": [0,1,2,3,4,5,6], "person1": ["ryan", "delta", "delta", "delta","bravo","alpha","ryan"], "person2": ["delta", "ryan", "alpha", "bravo","delta","ryan","alpha"], "messages": [1, 1, 2, 3,3,9,9]})   df person1 person2 messages 0 0 ryan delta 1 1 1 delta ryan 1 2 2 delta alpha 2 3 3 delta bravo 3 4 4 bravo delta 3 5 5 alpha ryan 9 6 6 ryan alpha 9  Answer df should be:  finaldf person1 person2 messages 0 0 ryan delta 1 1 2 delta alpha 2 2 3 delta bravo 3 3 5 alpha ryan 9 

Answer: Try as follows: res = (df[~df.filter(like='person').apply(frozenset, axis=1).duplicated()] .reset_index(drop=True)) print(res) person1 person2 messages 0 0 ryan delta 1 1 2 delta alpha 2 2 3 delta bravo 3 3 5 alpha ryan 9  Explanation  First, we use df.filter to select just the columns with person*. For these columns only we use df.apply to turn each row (axis=1) into a frozenset. So, at this stage, we are looking at a pd.Series like this:  0 (ryan, delta) 1 (ryan, delta) 2 (alpha, delta) 3 (bravo, delta) 4 (bravo, delta) 5 (alpha, ryan) 6 (alpha, ryan) dtype: object   Now, we want to select the duplicate rows, using Series.duplicated and add ~ as a prefix to the resulting boolean series to select the inverse from the original df. Finally, we reset the index with df.reset_index.
Question: I am trying to annotate an Iterator which only returns two values, T and cls[T]. Currently I have it annotated like this: from __future__ import annotations import typing class Node(typing.Generic[T]): def __init__(self, value: T, next: typing.Optional[Node[T]] = None) -> None: self.value = value self.next = next def __iter__(self) -> typing.Iterator[typing.Union[T, Node[T]]]: yield from (self.value, self.next)  This code works without any errors. However, I wish to take advantage of the fact that this dunder method will always yield two values to simplify the process for the user. As it is, the user has to deal with the following: one = Node[int](1, Node[int](2)) value, next = one # value = 1, next = Node(2) (i.e one.next) # This is a typing error because next can be either int or Node[int] # and 'two' is expected to be Node[int]: two: Node[int] = next # we can fix this by doing type narrowing assert isinstance(next, Node) two = next # now the error should be gone  So basically I want to take advantage of the fact that the second thing returned by the __iter__ is always of type Node[T] to avoid having to do type narrowing. I know that I have to change typing.Union[T, Node[T]] in the method's return annotation, but I do not know what to change it to.

Answer: This is not possible to annotate using the generic Iterator. It is a class expecting exactly one type argument. (current typeshed source) That means every value returned by its __next__ method necessarily has the same type. You are invoking the iterator protocol on that tuple of self.value, self.next. A tuple has an arbitrary number of type arguments (see here), but an iterator over it must still have exactly one. This actually leads to typing issues fairly often. Since you seem to intend your Node class to essentially emulate the tuple interface, this may be one of the rare cases, where it is better to inherit from it directly. A tuple will obviously also give you the iterable protocol, so you can still unpack it as before, but the types should be inferred properly, if you do everything correctly. Here is a full working example: from __future__ import annotations from typing import TypeVar, Optional T = TypeVar("T") class Node(tuple[T, "Node[T]"]): def __new__(cls, value: T, next_: Optional[Node[T]] = None) -> Node[T]: return tuple.__new__(cls, (value, next_)) def __init__(self, value: T, next_: Optional[Node[T]] = None) -> None: self.value = value self.next = next_ if __name__ == "__main__": node_1 = Node(1, Node(2)) val: int node_2: Node[int] val, node_2 = node_1  This passes mypy --strict without problems.  As an unrelated side note, I would advise against using built-in names like next. Also, note that you do not need to specify the type argument for Node, when you initialize one because it is automatically bound to the type passed to the value parameter.
Question: There is a class attribute spark in our AnalyticsWriter class: class AnalyticsWriter: spark = SparkSession.getActiveSession() # this is not getting executed  I noticed that this code is not being executed before a certain class method is run. Note: it has been verified that there is already an active SparkSession available in the process: so the init code is simply not being executed  @classmethod def measure_upsert( cls ) -> DeltaTable: assert AnalyticsWriter.spark, "AnalyticsWriter requires \ an active SparkSession"  I come from jvm-land (java/scala) and in those places the class level initialization code happens before any method invocations. What is the equivalent in python?

Answer: Class attributes are initialized at the moment they are hit, during class definition, so the line containing the getActiveSession() call is run before the class is even fully defined. class AnalyticsWriter: spark = SparkSession.getActiveSession() # The code has been run here # ... other definitions that occur after spark exists ... # class is complete here  I suspect the code is doing something, just not what you expect. You can confirm that it is in fact run with a cheesy hack like: class AnalyticsWriter: spark = (SparkSession.getActiveSession(), print("getActiveSession called", flush=True))[0]  which just makes a tuple of the result of your call and an eager print, then discards the meaningless result from the print; you should see the output from the print immediately, before you can get around to calling class methods.
Question: Using sqlalchemy 1.4, I want to map a class to a table that has a UNIQUEIDENTIFIER primary key. Using sqlalchemy.String does not work (complains about the fact that you cannot increment it). Checking dialects, I tried to used sqlalchemy.dialects.mssql.UNIQUEIDENTIFIER, however this does not work either: class Result(Base): __tablename__ = os.environ["RESULT_TABLE_NAME"] __table_args__ = {"schema": "myschema"} id = Column(UNIQUEIDENTIFIER) sqlalchemy.exc.ArgumentError: Mapper mapped class Result->DT_ODS_RESULT could not assemble any primary key columns for mapped table 'DT_ODS_RESULT'  Using the primary key parameters: class Result(Base): __tablename__ = os.environ["RESULT_TABLE_NAME"] __table_args__ = {"schema": "SIDODS"} id = Column(UNIQUEIDENTIFIER, primary_key=True) sqlalchemy.orm.exc.FlushError: Instance <Result at 0x7fbf9ab99d60> has a NULL identity key. If this is an auto-generated value, check that the database table allows generation of new primary key values, and that the mapped Column object is configured to expect these generated values. Ensure also that this flush() is not occurring at an inappropriate time, such as within a load() event.  Which I do not understand, because my column has a default value, as shown in this query part: ALTER TABLE [SIDODS].[DT_ODS_RESULT] ADD DEFAULT (newid()) FOR [ID]  The row is inserted with the following: r = Result( query_id=query_id, trigger_id=trigger_id, insertion_datetime=get_datetime_at_timezone() ) session.add(r) session.commit()  How to correctly map my sqlalchemy model class so that I can insert, without specifying it manually, a row with a UNIQUEIDENTIFIER type ?

Answer: Specifying primary_key=True and server_default=text("newid()") is sufficient to let SQLAlchemy know that the server will take case of assigning the PK value: from sqlalchemy import Column, String, create_engine, select, text from sqlalchemy.dialects.mssql import UNIQUEIDENTIFIER from sqlalchemy.orm import Session, declarative_base engine = create_engine("mssql+pyodbc://scott:tiger^5HHH@mssql_199") Base = declarative_base() class Thing(Base): __tablename__ = "thing" id = Column(UNIQUEIDENTIFIER, primary_key=True, server_default=text("newid()")) description = Column(String(100)) Base.metadata.drop_all(engine, checkfirst=True) engine.echo = True Base.metadata.create_all(engine) """DDL emitted: CREATE TABLE thing ( id UNIQUEIDENTIFIER NOT NULL DEFAULT newid(), description VARCHAR(100) NULL, PRIMARY KEY (id) ) """ engine.echo = False with Session(engine) as sess: sess.add(Thing(description="a thing")) sess.commit() with engine.begin() as conn: print(conn.execute(select(Thing.__table__)).all()) # [('32570502-7F18-40B5-87E2-7A60232FE9BD', 'a thing')]
Question: How do I add this: -param:abc=def as a SINGLE command line argument? Module subprocess splits this up in TWO arguments by replacing the equal sign with a space. Here is my python script: import subprocess pa=['test.bat', '--param:abc=def'] subprocess.run(pa)  Here is the test program test.bat: @echo off echo Test.bat echo Arg0: %0 echo Arg1: %1 echo Arg2: %2 pause  and here the output: Test.bat Arg0: test.bat Arg1: --param:abc Arg2: def Press any key to continue . . .  Because the equal sign is gone, the real app will not be started correctly. By the way, this problem also seems to happen when running on linux, with a sh script instead of a bat file. I understand that removing the equal sign is a 'feature' in certain cases, e.g. with the argparse module, but in my case I need to keep the equal sign. Any help is appreciated!

Answer: Welcome to .bat file hell To preserve equal sign, you'll have to quote your argument (explained here Preserving "=" (equal) characters in batch file parameters) ('"--param:abc=def"'), but then subprocess will escape the quotes Test.bat Arg0: test.bat Arg1: \"--param:abc=def\" Arg2:  Good old os.system won't do that import os os.system('test.bat "--param:abc=def"')  result Test.bat Arg0: test.bat Arg1: "--param:abc=def" Arg2:  Damn, those quotes won't go off. Let's tweak the .bat script a little to remove them manually @echo off echo Test.bat echo Arg0: %0 rem those 2 lines remove the quotes set ARG1=%1 set ARG1=%ARG1:"=% echo Arg1: %ARG1% echo Arg2: %2  now it yields the proper result Test.bat Arg0: test.bat Arg1: --param:abc=def Arg2:  Alternatively, stick to subprocess and remove quotes AND backslashes.