eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I tried to make python str generator with condition, but it is not easy to me. condition is simple.  I have 3 letters, "A", "B", and "C" each letters must be used at least 4 times. total sentence length is 19. I need to test all combinations.  so i tried below code for i in combinations_with_replacement('ABC', 7): for j in permutation(i+("A","B","C","A","B","C","A","B","C","A","B","C",), 19): test j  I think this code covers all combinations but it contains duplicants. how can I make it better?

Answer: To implement the rule that "each letters must be used at least 4 times" you can create a base pool of 12 characters with 'ABC' repeated 4 times, and that leaves 19 - 3 x 4 = 7 characters that need to be filled in with any of the letters in 'ABC', which can be done with itertools.combinations_with_replacement. Chain the base pool of 12 characters and the additional pool of 7 characters together to generate distinct permutations with more_itertools.distinct_permutations: from itertools import chain, combinations_with_replacement from more_itertools import distinct_permutations def string_generator(letters='ABC', min_count=4, length=19): base = letters * min_count for extra in combinations_with_replacement(letters, length - len(base)): yield from map(''.join, distinct_permutations(chain(base, extra)))  Demo: https://replit.com/@blhsing1/SandyNimblePatches
Question: I have a dataframe with two dates, among other things. I need to filter out rows that have more than two working days difference between these two dates. I must take into consideration weekends and holidays. *Assuming 10/17/2023 is a holiday... Example df: NAME DATE1 DATE2 CASE1 10/12/2023 10/13/2023 <--- one day difference CASE2 10/12/2023 10/16/2023 <--- two days difference (weekend) CASE3 10/12/2023 10/18/2023 <--- three days difference (weekends and holidays) ... CASEX 10/12/2023 10/19/2023 <--- four days difference (weekends and holidays)  I need to save CASE3 and CASEX (which has more than two days difference) in another dataframe and delete it from this one. My approach: date1 = "10/12/2023" date2 = "10/19/2023" date1 = pd.to_datetime(date1, format="%m/%d/%Y").date() date2 = pd.to_datetime(date2, format="%m/%d/%Y").date() holidays = [pd.to_datetime("10/17/2023",format="%m/%d/%Y").date()] days = np.busday_count(date1, date2, holidays=holidays)  In "days" I have the correct number. But I don't get to implement it in dataframe to filter it and extract the rows.

Answer: Code Define a list of holidays holidays = np.array([pd.to_datetime("10/17/2023", format="%m/%d/%Y")], dtype='datetime64[D]')  Parse the strings in date columns to datetime type df['DATE1'] = pd.to_datetime(df['DATE1'], format="%m/%d/%Y") df['DATE2'] = pd.to_datetime(df['DATE2'], format="%m/%d/%Y") # NAME DATE1 DATE2 # 0 CASE1 2023-10-12 2023-10-13 # 1 CASE2 2023-10-12 2023-10-16 # 2 CASE3 2023-10-12 2023-10-18 # 3 CASEX 2023-10-12 2023-10-19  Cast the dates to datetime64[D] types then use busy_day count to get the diff days = np.busday_count(df['DATE1'].values.astype("datetime64[D]"), df['DATE2'].values.astype("datetime64[D]"), holidays=holidays) # array([1, 2, 3, 4])  Use boolean indexing to filter the rows valid_rows = df[days <= 2] invalid_rows = df[days > 2] # valid_rows # NAME DATE1 DATE2 # 0 CASE1 2023-10-12 2023-10-13 # 1 CASE2 2023-10-12 2023-10-16 # invalid_rows # NAME DATE1 DATE2 # 2 CASE3 2023-10-12 2023-10-18 # 3 CASEX 2023-10-12 2023-10-19
Question: I want to load a huggingface pretrained transformer model directly to GPU (not enough CPU space) e.g. loading BERT from transformers import AutoModelForCausalLM model = AutoModelForCausalLM.from_pretrained("bert-base-uncased")  would be loaded to CPU until executing model.to('cuda')  now the model is loaded into GPU I want to load the model directly into GPU when executing from_pretrained. Is this possible?

Answer: I'm answering my own question. Hugging Face accelerate (add via pip install accelerate) could be helpful in moving the model to GPU before it's fully loaded in CPU. It's useful when: GPU memory > model size > CPU memory Also specify device_map="cuda": from transformers import AutoModelForCausalLM model = AutoModelForCausalLM.from_pretrained("bert-base-uncased", device_map="cuda")
Question: What the difference between using the Python typing.cast method x = cast(str, x)  compared with using type hints/colon notation/left hand side type annotation? x: str  I've seen the use of the cast method in codebases but don't have an apparent reason to use it instead of type hint notation, which is more concise.

Answer: I will answer you through a well-written example: def get_name(can_return_name: bool) -> str | None: in_name = input('Enter your name: ') return in_name if can_return_name else None def do_something_with_name() -> None: name: str = get_name(True) print(name.capitalize())  Here, you'd expect Pyright or mypy to say that name's type is str, even if the return type of get_name is str | None, but this is not the case: if the function you are calling (in this case get_name) has a return type, the column notation type will be ignored completely, and Pyright will tell you that you can't call .capitalize on a name, as it could be None In this case though, you are completely sure that name will be str and not None, so you can use cast, like such: def do_something_with_name() -> None: name: str = cast(str, get_name(True)) print(name.capitalize())  get_name's value will be unchanged, but static type checkers will treat them as the first argument's type. in fact, as python's documentation states, the cast function does in fact nothing at all. It can be defined with the new Python 3.12 generics as such: def cast[T](var: object) -> T: return var cast[str](get_name(True))
Question: I am currently exploring the massive Common Crawl dataset hosted on Amazon S3 and am attempting to use Amazon Athena to query this dataset. My objective is to search within the HTML content of the web pages to identify those that contain specific strings within their tags. Essentially, I am looking to filter out websites whose HTML content matches particular criteria. I am aware that Athena is capable of querying large datasets on S3 using standard SQL. However, I am not entirely sure about the feasibility and the approach to directly query inside the HTML content of the web pages in the Common Crawl dataset. Here's a simplified version of what I am looking to achieve: sql SELECT * FROM "common_crawl_dataset" WHERE html_content LIKE '%specific-string%';  Is it possible to directly query the HTML content of the web pages in the Common Crawl dataset using Athena? If yes, what would be the best approach to accomplish this, considering efficiency and cost-effectiveness? Are there any limitations or challenges that I should be aware of?

Answer: This is not easily possible, because the html content is not in the schema of the index that you are querying. Please see the Common Crawl Columnar Index blog post for further details. The most common use of this index is to select a small subset of the crawl (things like "all webpages with a Swiss domain name (*.ch) classified as being in the Romansh "roh" language). Accessing the html for these selected web captures is a second step. There is a large list of examples of columnar index (in many programming languages) in the cc-index-table GitHub repo.
Question: I have two dataframes that can both be empty, and I want to concat them. Before I could just do : output_df= pd.concat([df1, df2])  But now I run into  FutureWarning: The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.  An easy fix would be: if not df1.empty and not df2.empty: result_df = pd.concat([df1, df2], axis=0) elif not df1.empty: result_df = df1.copy() elif not df2.empty: result_df = df2.copy() else: result_df = pd.DataFrame()  But that seems pretty ugly. Does anyone have a better solution ? FYI: this appeared after pandas released v2.1.0

Answer: To be precise, concat is not deprecated (and won't be IMHO) but I can trigger this FutureWarning in 2.1.1 with the following example, while df2 being an empty DataFrame with a different dtypes than df1 : df1 = pd.DataFrame({"A": [.1, .2, .3]}) df2 = pd.DataFrame(columns=["A"], dtype="object") out = pd.concat([df1, df2]) print(out) A 0 0.1 1 0.2 2 0.3  As a solution in your case, you can try something like you did : out = (df1.copy() if df2.empty else df2.copy() if df1.empty else pd.concat([df1, df2]) # if both DataFrames non empty )  Or maybe even this one? : out = pd.concat([df1.astype(df2.dtypes), df2.astype(df1.dtypes)])
Question: so im new to testing and have been learning python/playwright testing for a couple of weeks now. I've got to assertions and have found multiple ways to write them, so this might sound like a stupid question but which of these 3 types of assertions would be best to use, or is there one that i don't know about that is better. Playwright Assertions: https://playwright.dev/python/docs/test-assertions Unittest Assertions: https://docs.python.org/3/library/unittest.html Python Assertions: https://realpython.com/python-assert-statement/ Here is a little example: class ExperimentTest(StaticLiveServerTestCase): def test_experimental(self): page.goto(f"{self.live_server_url}") expect(page).to_have_url(f"{self.live_server_url}") assert page.url == f"{self.live_server_url}") self.assertEqual(page.url, f"{self.live_server_url}") 

Answer: When using Playwright, almost always use the Playwright assertion, which waits for the predicate to be true. Consider the following simple example: from playwright.sync_api import expect, sync_playwright html = r"""<!DOCTYPE html><html><body> <h1>NO!</h1> <script> setTimeout(() => { document.querySelector("h1").textContent = "YES!" }, 3000); </script> </body></html> """ def main(): with sync_playwright() as p: browser = p.chromium.launch() page = browser.new_page() page.set_content(html) expect(page.locator("h1")).to_have_text("YES!") browser.close() if __name__ == "__main__": main()  This passes, even though the site takes 3 seconds after load to change its header text from NO! to YES!. If you use any other assertion library, the auto-wait will not occur. Now change the above code to fail: expect(page.locator("h1")).to_have_text("NEVER!"). The output will be clear: AssertionError: Locator expected to have text 'NEVER!' Actual value: YES! Call log: LocatorAssertions.to_have_text with timeout 5000ms waiting for locator("h1") locator resolved to <h1>NO!</h1> unexpected value "NO!" locator resolved to <h1>NO!</h1> unexpected value "NO!" locator resolved to <h1>NO!</h1> unexpected value "NO!" locator resolved to <h1>NO!</h1> unexpected value "NO!" locator resolved to <h1>NO!</h1> unexpected value "NO!" locator resolved to <h1>NO!</h1> unexpected value "NO!" locator resolved to <h1>NO!</h1> unexpected value "NO!" locator resolved to <h1>YES!</h1> unexpected value "YES!" locator resolved to <h1>YES!</h1> unexpected value "YES!"  Notice it's retried and tracked changes over time, eventually emitting a clear error which makes debugging easier. As an aside, there's no need to use f-strings if you're not concatenating anything: f"{self.live_server_url}" should be self.live_server_url.
Question: Today I installed the new Python 3.12 on my Ubuntu 22.04 from the ppa repository ppa:deadsnakes/ppa. Everything works, but when I try to install some library with the command python3.12 -m pip install somelibrary, I get the following error ERROR: Exception: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/pip/_internal/cli/base_command.py", line 165, in exc_logging_wrapper status = run_func(*args) ^^^^^^^^^^^^^^^ File "/usr/lib/python3/dist-packages/pip/_internal/cli/req_command.py", line 205, in wrapper return func(self, options, args) ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3/dist-packages/pip/_internal/commands/install.py", line 285, in run session = self.get_default_session(options) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3/dist-packages/pip/_internal/cli/req_command.py", line 75, in get_default_session self._session = self.enter_context(self._build_session(options)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3/dist-packages/pip/_internal/cli/req_command.py", line 89, in _build_session session = PipSession( ^^^^^^^^^^^ File "/usr/lib/python3/dist-packages/pip/_internal/network/session.py", line 282, in __init__ self.headers["User-Agent"] = user_agent() ^^^^^^^^^^^^ File "/usr/lib/python3/dist-packages/pip/_internal/network/session.py", line 157, in user_agent setuptools_dist = get_default_environment().get_distribution("setuptools") ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3/dist-packages/pip/_internal/metadata/__init__.py", line 24, in get_default_environment from .pkg_resources import Environment File "/usr/lib/python3/dist-packages/pip/_internal/metadata/pkg_resources.py", line 9, in <module> from pip._vendor import pkg_resources File "/usr/lib/python3/dist-packages/pip/_vendor/pkg_resources/__init__.py", line 2164, in <module> register_finder(pkgutil.ImpImporter, find_on_path) ^^^^^^^^^^^^^^^^^^^ AttributeError: module 'pkgutil' has no attribute 'ImpImporter'. Did you mean: 'zipimporter'?  Any suggestions why this is happening? EDIT: This problem doesn't exist when I use venv, it seems to me that the problem is that pip uses /usr/lib/python3 instead of /usr/lib/python3.12

Answer: python3.12 -m ensurepip --upgrade fixed my problem! solution
Question: I use Django framework to create basic web application. I started to write tests for my views. I followed the django documentation and fixed some issues along the way. But now I am stuck - I don't know why I get this error even after 30 minutes of searching for answer.  C:\Osobni\realityAuctionClient\venv\Scripts\python.exe "C:/Program Files/JetBrains/PyCharm 2022.3.3/plugins/python/helpers/pycharm/django_test_manage.py" test realityAuctionClientWeb.test_views.TestViews.test_start_view C:\Osobni\realityAuctionClient Testing started at 7:41 ... Traceback (most recent call last): File "C:\Program Files\JetBrains\PyCharm 2022.3.3\plugins\python\helpers\pycharm\django_test_manage.py", line 168, in <module> utility.execute() File "C:\Program Files\JetBrains\PyCharm 2022.3.3\plugins\python\helpers\pycharm\django_test_manage.py", line 142, in execute _create_command().run_from_argv(self.argv) File "C:\Osobni\realityAuctionClient\venv\Lib\site-packages\django\core\management\commands\test.py", line 24, in run_from_argv super().run_from_argv(argv) File "C:\Osobni\realityAuctionClient\venv\Lib\site-packages\django\core\management\base.py", line 412, in run_from_argv self.execute(*args, **cmd_options) File "C:\Osobni\realityAuctionClient\venv\Lib\site-packages\django\core\management\base.py", line 458, in execute output = self.handle(*args, **options) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Program Files\JetBrains\PyCharm 2022.3.3\plugins\python\helpers\pycharm\django_test_manage.py", line 104, in handle failures = TestRunner(test_labels, **options) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Program Files\JetBrains\PyCharm 2022.3.3\plugins\python\helpers\pycharm\django_test_runner.py", line 254, in run_tests return DjangoTeamcityTestRunner(**options).run_tests(test_labels, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Program Files\JetBrains\PyCharm 2022.3.3\plugins\python\helpers\pycharm\django_test_runner.py", line 156, in run_tests return super(DjangoTeamcityTestRunner, self).run_tests(test_labels, extra_tests, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ TypeError: DiscoverRunner.run_tests() takes 2 positional arguments but 3 were given Process finished with exit code 1 Empty suite   My tests.py looks like this:  import django from django.test import TestCase from django.urls import reverse class TestViews(TestCase): def test_start_view(self): """ Test that the start view returns a 200 response and uses the correct template """ django.setup() url = reverse("start") resp = self.client.get(url) self.assertEqual(resp.status_code, 200)  I use python 3.11 and django 5.0a1. Anybody who has a clue? I think it is somehow connected with settings but I do not know how.

Answer: Django removed the parameter extra_tests in Django 5.0 and PyCharm's test runner is still providing it. See django's release notes:  The extra_tests argument for DiscoverRunner.build_suite() and DiscoverRunner.run_tests() is removed.  You could downgrade to django 4.2 until this is fixed by Jetbrains. A Pycharm issue regarding this deprecation exists, see PY-53355.
Question: I am using Llama to create an application. Previously I used openai but am looking for a free alternative. Based on my limited research, this library provides openai-like api access making it quite easy to add into my prexisting code. However this library has errors while downloading. I tried installing cmake which did not help. Building wheels for collected packages: llama-cpp-python Building wheel for llama-cpp-python (pyproject.toml) ... error error: subprocess-exited-with-error × Building wheel for llama-cpp-python (pyproject.toml) did not run successfully. │ exit code: 1 ╰─> [20 lines of output] *** scikit-build-core 0.5.1 using CMake 3.27.7 (wheel) *** Configuring CMake... 2023-10-10 21:23:02,749 - scikit_build_core - WARNING - Can't find a Python library, got libdir=None, ldlibrary=None, multiarch=None, masd=None loading initial cache file C:\Users\ARUSHM~1\AppData\Local\Temp\tmpf1bzj6ul\build\CMakeInit.txt -- Building for: NMake Makefiles CMake Error at CMakeLists.txt:3 (project): Running 'nmake' '-?' failed with: The system cannot find the file specified CMake Error: CMAKE_C_COMPILER not set, after EnableLanguage CMake Error: CMAKE_CXX_COMPILER not set, after EnableLanguage -- Configuring incomplete, errors occurred! *** CMake configuration failed [end of output] note: This error originates from a subprocess, and is likely not a problem with pip. ERROR: Failed building wheel for llama-cpp-python Failed to build llama-cpp-python ERROR: Could not build wheels for llama-cpp-python, which is required to install pyproject.toml-based projects  Although not directly related to this question, these are other questions I am unable to get answers for:  Does this library use Llama or Llama 2? Will this be secure on a Python Flask Application? 

Answer: You need to install the desktop c++ block with visual studio to get cmake properly installed.Open the Visual Studio Installer and click Modify, then check Desktop development with C++ and click Modify to start the install. I also recommend the Windows 10 SDK. https://learn.microsoft.com/en-us/cpp/build/cmake-projects-in-visual-studio?view=msvc-170 https://developer.microsoft.com/en-us/windows/downloads/windows-sdk/ After that, !pip install llama-cpp-python should build just fine.
Question: Suppose I want a validator for checksums that I can reuse throughout an application. The Python type-hint system has changed a lot in 3.9+ and that is adding to my confusion. In pydantic v1, I subclassed str and implement __get_pydantic_core_schema__ and __get_validators__ class methods. v2 has changed some of this and the preferred way has changed to using annotated types. In pydantic-extra-types package I have found examples that require insight into the inner workings of pydantic. I can copy and get something working, but I would prefer to find the "correct" user's way to do it rather than copying without understanding. In pydantic v2 it looks like I can make a constrained string as part of a pydantic class, e.g. from typing import Annotated from pydantic import BaseModel, StringConstraints class GeneralThing(BaseModel): special_string = Annotated[str, StringConstraints(pattern="^[a-fA-F0-9]{64}$")]  but this is not valid (pydantic.errors.PydanticUserError: A non-annotated attribute was detected). Additionally I would have to annotate every field I want to constrain, as opposed to special_string = ChecksumStr that I was able to do in the past.

Answer: For pydantic you need to annotate your fields, but you're assigning them. The following code should do the trick for you from typing import Annotated from pydantic import BaseModel, StringConstraints ChecksumString = Annotated[str, StringConstraints(pattern="^[a-fA-F0-9]{64}$")] class GeneralThing(BaseModel): special_string: ChecksumString  Note that special_string: ChecksumString and not special_string = ChecksumString. special_string: Annotated[str, StringConstraints(...) would be valid as well just to know.
Question: I've installed scikit-fuzzy but when I import skfuzzy as fuzz I get an error ModuleNotFoundError: No module named 'distutils'"  I already tried to pip uninstall distutils and got this output Note: you may need to restart the kernel to use updated packages. WARNING: Skipping distutils as it is not installed.  Then I tried to install it again pip install distutils Note: you may need to restart the kernel to use updated packages. ERROR: Could not find a version that satisfies the requirement distutils (from versions: none) ERROR: No matching distribution found for distutils  Where did I go wrong?  This question addresses the problem from the perspective of installing a library. For developing a library, see How can one fully replace distutils, which is deprecated in 3.10?.

Answer: Python 3.12 does not come with a stdlib distutils module (changelog), because distutils was deprecated in 3.10 and removed in 3.12. See PEP 632 – Deprecate distutils module. You can still use distutils on Python 3.12+ by installing setuptools. When that doesn't work, you may need stay on Python < 3.12 until the 3rd-party package (skfuzzy in this case) publishes an updated release for Python 3.12 support.
Question: I have a situation where I need to run the adb shell command which takes me to the root access of my device. Then I run the interactive console app against my device using this command: /oem/console --no-logging. The command returns this output:  Console Process: connecting to application... Console Process: connected to application...  After this, I need to PRESS ENTER so that the console accepts events (The arrow cursor is ready to accept invoke events). This console app takes certain events that I can simulate my device's behavior. For example:  invoke put.device.into.charge 1 invoke call.cell.hangup 1 invoke xyz  So my question is, how do I one-by-one enter these commands into this console app and wait a couple of seconds after each command? This is the code I have so far: def run_adb_shell_command(command, return_process=False): """Run an ADB shell command.""" cmd = command.split() process = Popen(cmd, stdin=subprocess.PIPE) if return_process: return process def send_command_to_console_app(command, process): """SEND Commands to Console App.""" new_line = '\n'.encode() # encoded_command = str(command).encode() s0 = command s1 = 'event 1' s2 = 'event 2' s3 = 'event 3' concat_query = "{}\n".format(s0) process.communicate(input=concat_query.encode('utf-8'))[0] process.kill()  The problem is that when communicate() function is called for one of the events it just hangs up and never finishes so that I can run the next event. What is the best approach for this?

Answer: So, after a couple of days of research, I found a solution from this post https://stackoverflow.com/a/56051270/4911426. I had to use an asynchronous subprocess for chaining different commands together.