eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I am trying to install a NLP suite on my macbook pro, which is updated to the most recent software version Catalina 10.15.6. So far, I have installed Anaconda 3.8, created a version 3.7 NLP environment by conda create -n NLP python=3.7, and activated the NLP environment by conda activate NLP. My next step is to install all python packages that are written in the file "requirements.txt" with the following command pip install -r requirements.txt. However, it showcases this message: "ERROR: Could not find a version that satisfies the requirement pprint (from -r requirements.txt (line 67)) (from versions: none) ERROR: No matching distribution found for pprint (from -r requirements.txt (line 67))" I also tried installing the package alone, however, the same error message appears. Any advice would be appreciated! Please let me know if any additional information I can provide.

Answer: pprint is part of the standard library, therefore cannot be present in requirements.txt. If one of your requirements is stated to require pprint you'll get an error. To install without dependencies use the --no-deps command for pip. However, this does not guarantee that the installation actually worked as you are likely missing out on other packages. So a better option is installing each requirement one by one until you find the one that needs it and install its other dependencies and install that package with no-deps. An alternative is to use https://pypi.org/project/pipdeptree/ to inspect the dependency tree. If there are many packages and there is a version freeze, try dropping the versions. It is a bit of trial and error detective work, so one can be smart about it: it is likely a less used dependency that is the culprit.
Question: I am seeing this for the first time. I wanted to know what the !r in the last line of the code called so that I can search about it. I found this piece of code on: https://adamj.eu/tech/2020/08/10/a-guide-to-python-lambda-functions/ class Puppy: def __init__(self, name, cuteness): self.name = name self.cuteness = cuteness def __repr__(self): return f"Puppy({self.name!r}, {self.cuteness!r})" 

Answer: It's a format string conversion flag that tells the formatter to call repr on the object before formatting the string.  Three conversion flags are currently supported: '!s' which calls str() on the value, '!r' which calls repr() and '!a' which calls ascii().
Question: Is there a method, or an attribute in request object that I can access to return me the URL exactly as the client requested? With the query params included? I've checked request.build_absolute_uri after looking at this question but it just returns the URL without the query params. I need the URL because my API response returns the URL for the "next page" of results. I could build it from the query_params attributes, but this view takes a lot of query params and some exclude others, so having access to the request url would save me a lot of pain.

Answer: To get full path, including query string, you want request.get_full_path()
Question: How can I dynamically create a subclass of my class and provide arguments to its __init_subclass__() method? Example class: class MyClass: def __init_subclass__(cls, my_name): print(f"Subclass created and my name is {my_name}")  Normally I'd implement my subclass as such: class MySubclass(MyClass, my_name="Ellis"): pass  But how would I pass in my_name when dynamically creating a subclass of MyClass using a metaclass? Normally I could use type() but it doesn't have the option of providing my_name. MyDynamicSubclass = type("MyDynamicSubclass", (MyClass,), {}) 

Answer: The basic documentation for type does not mention that it accepts an unlimited number of keyword-only arguments, which you would supply through the keywords in a class statement. The only place this is hinted in is in the Data Model in the section Creating the class object:  Once the class namespace has been populated by executing the class body, the class object is created by calling metaclass(name, bases, namespace, **kwds) (the additional keywords passed here are the same as those passed to __prepare__).  Normally, you would not use this feature with type exactly because of __init_subclass__:  The default implementation object.__init_subclass__ does nothing, but raises an error if it is called with any arguments.  Since you have overriden the default implementation, you can create your dynamic class as MyDynamicSubclass = type("MyDynamicSubclass", (MyClass,), {}, my_name="Ellis")
Question: I am using keras with a custom loss function like below: def custom_fn(y_true, y_pred): # changing y_true, y_pred values systematically return mean_absolute_percentage_error(y_true, y_pred)  Then I am calling model.compile(loss=custom_fn) and model.fit(X, y,..validation_data=(X_val, y_val)..) Keras is then saving loss and val_loss in model history. As a sanity check, when the model finishes training, I am using model.predict(X_val) so I can calculate validation loss manually with my custom_fn using the trained model. I am saving the model with the best epoch using this callback: callbacks.append(ModelCheckpoint(path, save_best_only=True, monitor='val_loss', mode='min'))  so after calculating this, the validation loss should match keras' val_loss value of the best epoch. But this is not happening. As another attempt to figure this issue out, I am also doing this:  model.compile(loss=custom_fn, metrics=[custom_fn])  And to my surprise, val_loss and val_custom_fn do not match (neither loss or loss_custom_fn for that matter). This is really strange, my custom_fn is essentially keras' built in mape with the y_true and y_pred slightly manipulated. what is going on here? PS: the layers I am using are LSTM layers and a final Dense layer. But I think this information is not relevant to the problem. I am also using regularisation as hyperparameter but not dropout. Update Even removing custom_fn and using keras' built in mape as a loss function and metric like so: model.compile(loss='mape', metrics=['mape'])  and for simplicity, removing ModelCheckpoint callback is having the same effect; val_loss and val_mape for each epoch are not equivalent. This is extremely strange to me. I am either missing something or there is a bug in Keras code..the former might be more realistic.

Answer: This blog post suggests that keras adds any regularisation used in the training when calculating the validation loss. And obviously, when calculating the metric of choice no regularisation is applied. This is why it occurs with any loss function of choice as stated in the question. This is something I could not find any documentation on from Keras. However, it seems to hold up since when I remove all regularisation hyperparameters, the val_loss and val_custom_fn match exactly in each epoch. An easy workaround is to either use the custom_fn as a metric and save the best model based on the metric (val_custom_fn) than on the val_loss. Or else Loop through each epoch manually and calculate the correct val_loss manually after training each epoch. The latter seems to make more sense since there is no reason to include custom_fn both as a metric and as a loss function. If anyone can find any evidence of this in the Keras documentation that would be helpful.
Question: I want to optimize the django app and for this I would like to know How can I check if my query is hitting database or I am getting result/return value from cached version? For example: products = Products.objects.filter(product_name__icontains="natural") if not products.exist(): return Response(...) total_products = products.count() first_product = product.first()  I like to execute this in shell and want to check which line hits the database and which one just return result from cached version so I can write optimized queries in my view. I know about django-toolbar but I couldn't find if it supports things like this(does certain line hit database or result is from cached version).

Answer: Check the lenth of connection.queries in this way, from django.conf import settings settings.DEBUG = True from django.db import connection print(len(connection.queries)) # do something with the database products = Products.objects.filter(product_name__icontains="natural") print(len(connection.queries)) # and execute the print statement again and again total_products = products.count() print(len(connection.queries)) first_product = product.first() print(len(connection.queries)) Reference:  Get SQL query count during a Django shell session
Question: So I looked at a bunch of posts on Pyspark, Jupyter and setting memory/cores/executors (and the associated memory). But I appear to be stuck - Question 1: I dont see my machine utilizing either the cores or the memory. Why? Can I do some adjustments to the excutors/cores/memory to optimize speed of reading the file? Question 2: Also is there any way for me to see a progress bar showing how much of the file ahs been imported (spark-monitor doesnt seem to do it). I am importing a 33.5gb file into pyspark. Machine has 112 gb or RAM 8 Cores/16 virtual cores. from pyspark.sql import SparkSession  spark = SparkSession \ .builder \ .appName("Summaries") \ .config("spark.some.config.option", "some-value") \ .getOrCreate() conf = spark.sparkContext._conf.setAll([('spark.executor.memory', '4g'), ('spark.app.name', 'Spark Updated Conf'), ('spark.driver.cores', '4'), ('spark.executor.cores', '16'), ('spark.driver.memory','90g')]) spark.sparkContext.stop() spark = SparkSession.builder.config(conf=conf).getOrCreate() df = spark.read.json("../Data/inasnelylargefile.json.gz")  I assume that the pyspark is doing its magic even while reading a file (so I should see heavy core/memory utilization). But I am not seeing it.Help! Update: Tested with a smaller zip file (89 MB) Pyspark take 72 seconds Pandas takes 10.6 seconds Code used : start = time.time() df = spark.read.json("../Data/small.json.gz") end = time.time() print(end - start) start = time.time() df = pa.read_json('../Data/small.json.gz',compression='gzip', lines = True) end = time.time() print(end - start) 

Answer: spark.sparkContext.stop() spark = SparkSession.builder.config(conf=conf).getOrCreate() df = spark.read.json("../Data/inasnelylargefile.json.gz")  Add this: df.show() ##OR df.persist()  The comparison you are doing is not apples to apples, spark performs lazy evaluation, meaning if you don't call an action over your operation, it will do nothing but just compile and keep the DAG ready for you. In Spark, there are two concepts,  Transformation: Evaluated lazily Actions: (like collect(), take(), show(),persist()) evaluated instantly.  In your case, read() is just a transformation, adding an action should trigger the computation. More about actions vs transformation: https://training.databricks.com/visualapi.pdf
Question: I am creating a python script that should modify itself and be portable. I can achieve each one of those goals separately, but not together. I use cx_freeze or pyinstaller to pack my .py to exe, so it's portable; but then I have a lot of .pyc compiled files and I can't edit my .py file from the software itself. Is there a way to keep a script portable and lightweight (so a 70mb portable python environment is not an option) but still editable? The idea is to have a sort of exe "interpreter" like python.exe but with all the libraries linked, as pyinstaller allows, that runs the .py file, so the .py script can edit itself or be edited by other scripts and still be executed with the interpreter.

Answer: First define your main script (cannot be changed) main_script.py. In a subfolder (e.g. named data) create patch_script.py main_script.py: import sys sys.path.append('./data') import patch_script  inside the subfolder: data\patch_script.py: print('This is the original file') In the root folder create a spec file e.g. by running pyinstaller main_script.py. Inside the spec file, add the patch script as a data resource:  ... datas=[('./data/patch_script.py', 'data' ) ], ...  Run pyinstaller main_sript.spec. Execute the exe file, it should print This is the original file Edit the patch script to e.g. say: print('This is the patched file') Rerun the exe file, it should print This is the patched file Note: As this is a PoC, this works but is prone to security issues, as the python file inside the data directory can be used for injection of arbitrary code (which you don't have any control of). You might want to consider using proper packages and update scripts as used by PIP etc.
Question: Intellisense: not working with conda (above), working fine when normal Python (below) As shown above, Intellisense does not work in VS Code when Conda Environment is set as Python interpreter, it is just keeps “Loading…”. When normal Python interpreter is set (that comes when installing Python extension), Intellisense is working fine. There are no problems to run or debug files with both environment, only issues is Intellisense in Conda Environment. I have tried at least following things without any success.  Restart VS Code several times Uninstalled and re-installed Anaconda Extension Pack (which installs Python extension) Run Python “Build Workspace Symbols” as suggested in How to enable intellisense for python in Visual Studio Code with anaconda3? I also tried command conda init powershell Opening VS Code from Anaconda Prompt as suggested in vscode IntelliSense / code completion doesn't work when I am not in base conda environment  System info: Version: 1.47.3 (user setup) Commit: 91899dcef7b8110878ea59626991a18c8a6a1b3e Date: 2020-07-23T13:12:49.994Z Electron: 7.3.2 Chrome: 78.0.3904.130 Node.js: 12.8.1 V8: 7.8.279.23-electron.0 OS: Windows_NT x64 10.0.16299

Answer: I found a similar problem, they solve it through explicitly set the python.pythonPath, you can refer to this page. In your problem, only when selecting the conda interpreter the Intellisense not work as the Intellisense was provided by the Language Server, Could you try these?  Select a different Language Server, The Language Server includes: 'Jedi'(build-in Python extension ), 'Microsoft', 'Pylance'(need install Pylance extension).  downgrade or upgrade the 'Python' extension.   If it still not work, you can try these to find more information which will be helpful to solve the problem:  Look in the OUTPUT panel, select the 'Python Language Server' channel to check whether the Language Server works well.  Open Help -> Toggle Developer Tools select the Console panel to take a check.
Question: In Prefect, suppose I have some pipeline which runs f(date) for every date in a list, and saves it to a file. This is a pretty common ETL operation. In airflow, if I run this once, it will backfill for all historical dates. If I run it again, it will know that the task has been run, and only run any new tasks that have appeared (ie latest date). In Prefect, to my knowledge, it will run the entire pipeline every day, even if 99% of the tasks were completed the day before. What are some solutions to dealing with this, without switching to Prefect Cloud? Do you just do something like make every task cache it's completion in redis before exiting?

Answer: Prefect has many first-class ways of handling caching, depending on how much control you want. For every task, you can specify whether results should be cached, how long they should be cached, and how the cache should be invalidated (age, different inputs to the task, flow parameter values, etc.). The simplest way to cache a task is to use targets, which lets you specify that the task has a templatable side effect (usually a file in local or Cloud storage, but could be e.g. a database entry, redis key, or anything else). Before the task is run, it checks if the side effect exists and if it does, skips the run. For example, this task will write its result to a local file automatically templated with the task name and the current date: @task(result=LocalResult(), target="{task_name}-{today}") def get_data(): return [1, 2, 3, 4, 5]  As long as a matching file exists, the task won't re-run. Because {today} is part of the target name, that will implicitly cache the task's value for one day. You could also use a parameter in the template, like the backfill date, to replicate Airflow's behavior. For more control, you can use Prefect's full cache mechanism by setting cache_for, cache_validator, and cache_key on any task. If set, the task will finish in a Cached state instead of a Success state. When paired with a proper orchestration backend like Prefect Server or Prefect Cloud, the Cached state can be queried by future runs of the same task (or any task with the same cache_key). That future task will return the Cached state as its own result.
Question: I am using Selenium to Log In to an account. After Logging In I would like to save the session and access it again the next time I run the python script so I don't have to Log In again. Basically I want to chrome driver to work like the real google chrome where all the cookies/sessions are saved. This way I don't have to login to the website on every run. browser.get("https://www.website.com") browser.find_element_by_xpath("//*[@id='identifierId']").send_keys(email) browser.find_element_by_xpath("//*[@id='identifierNext']").click() time.sleep(3) browser.find_element_by_xpath("//*[@id='password']/div[1]/div/div[1]/input").send_keys(password) browser.find_element_by_xpath("//*[@id='passwordNext']").click() time.sleep(5) 

Answer: This is the solution I used: # I am giving myself enough time to manually login to the website and then printing the cookie time.sleep(60) print(driver.get_cookies()) # Than I am using add_cookie() to add the cookie/s that I got from get_cookies() driver.add_cookie({'domain': ''})  This may not be the best way to implement it but it's doing what I was looking for
Question: On Django new version 3.1, the settings file have some changes, and I came to ask how I have to proceed to set my static files? The way that I usually did doesn't work more. Last versions: import os BASE_DIR = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))  Version 3.1: from pathlib import Path BASE_DIR = Path(__file__).resolve(strict=True).parent.parent  I usually set my static files like this: STATIC_URL = '/static/' MEDIA_URL = '/media/' STATICFILES_DIRS = [ os.path.join(BASE_DIR, 'static') ] STATIC_ROOT = os.path.join(BASE_DIR, 'static_root') MEDIA_ROOT = os.path.join(BASE_DIR, 'media_root')  If I insert the import os will work, but is it the right practice? What is the best practice to set this? Thank you?

Answer: This change makes it a lot easier for you to define your STATIC and MEDIA variables. You don't even need to import os for this purpose and all you need is to add following codes to your settings.py: BASE_DIR = Path(__file__).resolve(strict=True).parent.parent # which shows the root directory of your project STATIC_ROOT = BASE_DIR / 'static' # is equal to os.path.join(BASE_DIR, 'static/') STATIC_URL = '/static/' MEDIA_ROOT = BASE_DIR / 'media' # is equal to os.path.join(BASE_DIR, 'media/') MEDIA_URL = '/media/'
Question: While installing TensorFlow on Windows having python 3.9 installed using the following command: pip install tensorflow  Following error occurred with the warning: WARNING: Failed to write executable - trying to use .deleteme logic ERROR: Could not install packages due to an OSError: [WinError 2] The system cannot find the file specified: 'c:\\python39\\Scripts\\pyrsa-decrypt.exe' -> 'c:\\python39\\Scripts\\pyrsa-decrypt.exe.deleteme'  How this can be resolved?

Answer: Run the same command using --user pip install --user package_name  or you can try to restart the terminal and run it as admin
Question: Is there a way to show pair-correlation values with seaborn.pairplot(), as in the example below (created with ggpairs() in R)? I can make the plots using the attached code, but cannot add the correlations. Thanks import numpy as np import seaborn as sns import matplotlib.pyplot as plt iris = sns.load_dataset('iris') g = sns.pairplot(iris, kind='scatter', diag_kind='kde') # remove upper triangle plots for i, j in zip(*np.triu_indices_from(g.axes, 1)): g.axes[i, j].set_visible(False) plt.show()  

Answer: If you use PairGrid instead of pairplot, then you can pass a custom function that would calculate the correlation coefficient and display it on the graph: from scipy.stats import pearsonr def reg_coef(x,y,label=None,color=None,**kwargs): ax = plt.gca() r,p = pearsonr(x,y) ax.annotate('r = {:.2f}'.format(r), xy=(0.5,0.5), xycoords='axes fraction', ha='center') ax.set_axis_off() iris = sns.load_dataset("iris") g = sns.PairGrid(iris) g.map_diag(sns.distplot) g.map_lower(sns.regplot) g.map_upper(reg_coef)
Question: Heres is my setup setup( name="`...", version="...", description=..., long_description_content_type="text/markdown", long_description=README, author="...", classifiers=[...], packages=["..."], include_package_data=True, )  I used the following command to package my project python setup.py sdist bdist_wheel  but when I run twine check dist/*  I get the following error Checking dist\Futshane_TBG-1.0.0-py3-none-any.whl: FAILED `long_description` has syntax errors in markup and would not be rendered on PyPI. line 9: Error: Unexpected indentation. warning: `long_description_content_type` missing. defaulting to `text/x-rst`. Checking dist\Futshane_TBG-1.0.0.tar.gz: FAILED `long_description` has syntax errors in markup and would not be rendered on PyPI. line 9: Error: Unexpected indentation. warning: `long_description_content_type` missing. defaulting to `text/x-rst`.  Why is it failing to identify the type provided, when I've obviously provided one?

Answer: I attempted to switch the order of the "long_description_content_type" and the "long_description" arguments and instead of assigning the description argument to a variable containing the description, I assigned it directly to the description. Doing so has resolved my issue setup( name="Futshane_TBG", version="1.0.0", description=""" The description of the package """, long_description_content_type="text/markdown", long_description=README, url="https://github.com/ElLoko233/Text-Based-Game-Package", author="Lelethu Futshane", classifiers=["License :: OSI Approved :: MIT License", "Programming Language :: Python :: 3", "Programming Language :: Python :: 3.8"], packages=["TBG"], include_package_data=True, )
Question: I am working on a kind of test framework that happens to have classes named TestSomething of course. And I realized my tests are failing because pytest sees these classes as "something I need to instantiate and run!", as soon as imported. And this absolutely won't work. import pytest from package import TestSomethingClass  Is it possible to import such class from a pytest test file directly? Or should I indirectly use those via a fixture maybe? The same problem applies to exceptions, as I would need to do something like with pytest.raises(TestsSomethingError): 

Answer: Explicitly Disable You're able to allow pytest to ignore this specific class by virtue of it starting with the word Test, by setting the __test__ flag to False within your conflicting class class TestSomethingClass(object): __test__ = False def test_class_something(self, object): pass  feature: https://github.com/pytest-dev/pytest/pull/1561  Configuration File We could also change the convention completely in your pytest.ini file and ignore all class names which start the word Example. But I don't advice this, as we could potentially encounter unintended consequences down the road. #pytest.ini file [pytest] python_classes = !Example   Pytest Conventions  test prefixed test functions or methods outside of class  test prefixed test functions or methods inside Test prefixed test classes (without an init method)   Source: https://docs.pytest.org/en/stable/customize.html