eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: How can I set the niceness for each process in a multiprocessing.Pool? I understand that I can increment niceness with os.nice(), but how do call it in the child process after creating the pool? If I call it in the mapped function it will be called every time the function executes, rather than once when the process is forked. import multiprocessing as mp NICENESS = 19 DATA = range(100000) def foo(bar): return bar * 2 pool = mp.Pool(100) # Somehow set niceness of each process to NICENESS pool.map(foo, DATA) 

Answer: What about using an initializer for that? https://docs.python.org/3.8/library/multiprocessing.html#multiprocessing.pool.Pool The function is called once when the pool is started so the os.nice() call in the initializer sets the niceness for the proces after that. I've added some additional statements to show that it works in your worker function but the os.nice() calls should obviously be removed since you want a static niceness value. import multiprocessing as mp import os NICENESS = 3 DATA = range(6) def foo(bar): newniceness = os.nice(1) # remove this print('Additional niceness:', newniceness) # remove this return bar * 2 def set_nicesness(val): # the initializer newval = os.nice(val) # starts at 0 and returns newvalue print('niceness value:', newval) pool = mp.Pool(3, initializer=set_nicesness, initargs=(NICENESS,)) # Somehow set niceness of each process to NICENESS pool.map(foo, DATA)  As you can see from the prints the niceness now starts at 3 (I've set this for NICENESS) and starts incrementing from there. Or as a useable snippet import multiprocessing as mp import os NICENESS = 3 def mp_function(bar: int) -> int: return bar * 2 if __name__ == '__main__': pool = mp.Pool(3, initializer=os.nice, initargs=(NICENESS,)) data = range(6) pool.map(mp_function, data)
Question: EDIT: Solved, if anyone comes across this python3.8 -m pip install python-dotenv worked for me.  I've tried reinstalling both dotenv and python-dotenv but I'm still getting the same error. I do have the .env file in the same directory as this script.  #bot.py import os import discord from dotenv import load_dotenv load_dotenv() token=os.getenv('DISCORD_TOKEN') client = discord.Client() @client.event async def on_ready(): print(f'{client.user} has connected to Discord!') client.run(token) 

Answer: in your installation manager if it's Ubuntu or Debian try: apt install python3-dotenv you can also try sudo pip3 install dotenv to install via pip. Whatever you do remember to include explicitly the missing 3 part. Debian/Ubuntu have separate packages and as of the present time python means python2 and python3 means python3 in their apt repositories. However, when it comes to your locally installed python binary on your system which python binary it defaults to using may vary depending on what /usr/bin/python is symlinked to on your system. Some systems it's symlinked to something like python2.7 and other's it may be something like python3.5. Similar issues exist with locally installed pip. Hence, why using the '3' is important when installing or searching for python packages
Question: I have been learning about NLP models and came across word embedding, and saw the examples in which it is possible to see relations between words by calculating their dot products and such. What I am looking for is just a dictionary, mapping words to their representative vectors, so I can play around with it. I know that I can build a model and train it and create my own map but I just want the already trained map as a python variable. 

Answer: You can try out Google's word2vec model trained with about 100 billion words from various news articles. An interesting fact about word vectors, w2v(king) - w2v(man) + w2v(woman) ≈ w2v(queen)
Question: The error that I am getting when I use import torchvision is this: Error Message "*Traceback (most recent call last): File "/Users/gokulsrin/Desktop/torch_basics/data.py", line 4, in <module> import torchvision ModuleNotFoundError: No module named 'torchvision'*"  I don't know what to do. I have tried changing the version of python from the native one to the one downloaded through anaconda. I am using anaconda as a package manager and have installed torch vision through anaconda as well as through pip commands. 

Answer: From PyTorch installing Docs you should follow these steps:  In Anaconda use this command: conda install pytorch torchvision cpuonly -c pytorch  In Pip use this command: pip3 install torch==1.3.1+cpu torchvision==0.4.2+cpu -f https://download.pytorch.org/whl/torch_stable.html   Note: If you have an enabled CUDA card you can change the cpuonly option to cudatoolkit=10.1 or cudatoolkit=9.2 After successfully installing the package you can import it with the command import torchvision and the output should look like this:  Otherwise, there is something wrong when you are downloading the package from the Internet
Question: def insert(self): conn = pyodbc.connect( 'Driver={SQL Server};' 'Server=DESKTOP-S0VG212\SQLEXPRESS;' 'Database=MovieGuide;' 'Trusted_Connection=yes;' ) cursor = conn.cursor()  Error occurs when executing the query but I don't know what's causing it.  cursor.execute('insert into Movies(MovieName,Genre,Rating,Username) values(?,?,?,?);', (self.moviename, self.moviegenre, self.ratebox, self.username)) conn.commit() 

Answer: I know my answer is late, but it can be useful to someone. SystemError: <class 'pyodbc.Error'> returned a result with an error set error appears when the query is wrong, make sure you are executing the correct query using SQL server query window then you can able to identify the problem. In the question, the semicolon should not come to the end of the query, if you still get an error, it might chances for the column to have some constraint issue. So follow the below method when you are facing this issue.  Execute one insert query in the SQL server query tab and identify the problem.
Question: I encountered a bug-like feature of setup.py where I am getting the Permission denied error regardless where I want to install the package without root privilege. I have a toy python package with a few tiny files, and there is no problem of building it. There is nothing special in the setup.py file. I will list one or two of them. setup ( name='pmsi', entry_points={ 'console_scripts': [ 'pmsi = pmsi.pmsi:main', ] }, ) sudo python3 setup.py install  Gave me no problem at all. I need to install this package to a particular place and have tried --user, --home, --prefix options; all gave me the same error message at the egg_info step. python3 setup.py install --user running install running bdist_egg running egg_info error: [Errno 13] Permission denied  It appears that the install process always tries to copy the egg_info to some system place where I don't have permission to write. I am not an expert on setup.py, there must be some default rule that I can overwrite either on the command line or setup.py. Or should I always install to system place as root (that seems to be a bad choice, what if you want to test before a system install). 

Answer: The reason for this particular difficulty is because I run sudo before in the package directory and it created some directories owned by root. Afterwards, I run as a regular user and got permission issues. The fix is ownership change. cd ~/lib/python3.8/site-packages sudo chown -R myuid:mygroup *  After the above action, the problem was resolved. The actual python lib dir may be different for different situation.
Question: I'm using selenium with python,now I want to locate an element by part of its id name,what can I do? For example,now I've already located a item by id name coption5 : sixth_item = driver.find_element_by_id("coption5")  Is there anyway I can locate this element only by using coption?

Answer: To find the element which you have located with: sixth_item = driver.find_element_by_id("coption5")  To locate this element only by using coption you can use can use either of the following Locator Strategies:  Using XPATH and starts-with(): sixth_item = driver.find_element_by_xpath("//*[starts-with(@id, 'coption')]")   Using XPATH and contains(): sixth_item = driver.find_element_by_xpath("//*[contains(@id, 'coption')]")   Using CSS_SELECTOR and ^ (wildcard of starts-with): sixth_item = driver.find_element_by_css_selector("[id^='coption']")   Using CSS_SELECTOR and * (wildcard of contains): sixth_item = driver.find_element_by_css_selector("[id*='coption']")     Reference You can find a detailed discussion on dynamic CssSelectors in:  How to get selectors with dynamic part inside using Selenium with Python? Java Selenium webdriver expression finding dynamic element by ccs that starts with and ends with How to click a dynamic link with in a drupal 8 website using xpath/css selector while automating through Selenium and Python Finding elements by CSS selector with ChromeDriver (Selenium) in Python
Question: Can it iterate through the Pyspark groupBy dataframe without aggregation or count? For example code in Pandas: for i, d in df2: mycode .... ^^ if using pandas ^^ Is there a difference in how to iterate groupby in Pyspark or have to use aggregation and count? 

Answer: At best you can use .first , .last to get respective values from the groupBy but not all in the way you can get in pandas. ex: from pyspark.sql import functions as f df.groupBy(df['some_col']).agg(f.first(df['col1']), f.first(df['col2'])).show()  Since their is a basic difference between the way the data is handled in pandas and spark not all functionalities can be used in the same way. Their are a few work arounds to get what you want like: for diamonds DataFrame: +---+-----+---------+-----+-------+-----+-----+-----+----+----+----+ |_c0|carat| cut|color|clarity|depth|table|price| x| y| z| +---+-----+---------+-----+-------+-----+-----+-----+----+----+----+ | 1| 0.23| Ideal| E| SI2| 61.5| 55.0| 326|3.95|3.98|2.43| | 2| 0.21| Premium| E| SI1| 59.8| 61.0| 326|3.89|3.84|2.31| | 3| 0.23| Good| E| VS1| 56.9| 65.0| 327|4.05|4.07|2.31| | 4| 0.29| Premium| I| VS2| 62.4| 58.0| 334| 4.2|4.23|2.63| | 5| 0.31| Good| J| SI2| 63.3| 58.0| 335|4.34|4.35|2.75| +---+-----+---------+-----+-------+-----+-----+-----+----+----+----+  You can use: l=[x.cut for x in diamonds.select("cut").distinct().rdd.collect()] def groups(df,i): import pyspark.sql.functions as f return df.filter(f.col("cut")==i) #for multi grouping def groups_multi(df,i): import pyspark.sql.functions as f return df.filter((f.col("cut")==i) & (f.col("color")=='E'))# use | for or. for i in l: groups(diamonds,i).show(2)  output : +---+-----+-------+-----+-------+-----+-----+-----+----+----+----+ |_c0|carat| cut|color|clarity|depth|table|price| x| y| z| +---+-----+-------+-----+-------+-----+-----+-----+----+----+----+ | 2| 0.21|Premium| E| SI1| 59.8| 61.0| 326|3.89|3.84|2.31| | 4| 0.29|Premium| I| VS2| 62.4| 58.0| 334| 4.2|4.23|2.63| +---+-----+-------+-----+-------+-----+-----+-----+----+----+----+ only showing top 2 rows +---+-----+-----+-----+-------+-----+-----+-----+----+----+----+ |_c0|carat| cut|color|clarity|depth|table|price| x| y| z| +---+-----+-----+-----+-------+-----+-----+-----+----+----+----+ | 1| 0.23|Ideal| E| SI2| 61.5| 55.0| 326|3.95|3.98|2.43| | 12| 0.23|Ideal| J| VS1| 62.8| 56.0| 340|3.93| 3.9|2.46| +---+-----+-----+-----+-------+-----+-----+-----+----+----+----+ ...  In Function groups you can decide what kind of grouping you want for the data. It is a simple filter condition but it will get you all the groups separately.
Question: I am training a convolutional network and I want to stop training once the validation error hits 90%. I thought about using EarlyStopping and setting baseline to .90 but then it stops training whenever the validation accuracy is below that baseline for given number of epochs(which is just 0 here). So my code is: es=EarlyStopping(monitor='val_acc',mode='auto',verbose=1,baseline=.90,patience=0) history = model.fit(training_images, training_labels, validation_data=(test_images, test_labels), epochs=30, verbose=2,callbacks=[es])  When I use this code my training stops after the first epoch with given results: Train on 60000 samples, validate on 10000 samples Epoch 1/30 60000/60000 - 7s - loss: 0.4600 - acc: 0.8330 - val_loss: 0.3426 - val_acc: 0.8787  What else can I try to stop my training once the validation accuracy hits 90% or above? Here is the rest of the code:  tf.keras.layers.Conv2D(64, (3,3), activation='relu', input_shape=(28, 28, 1)), tf.keras.layers.MaxPooling2D(2, 2), tf.keras.layers.Conv2D(32, (3,3), activation='relu', input_shape=(28, 28, 1)), tf.keras.layers.MaxPooling2D(2, 2), tf.keras.layers.Flatten(), tf.keras.layers.Dense(152, activation='relu'), tf.keras.layers.Dense(10, activation='softmax') ]) model.compile(optimizer=Adam(learning_rate=0.001),loss='sparse_categorical_crossentropy', metrics=['accuracy']) es=EarlyStopping(monitor='val_acc',mode='auto',verbose=1,baseline=.90,patience=0) history = model.fit(training_images, training_labels, validation_data=(test_images, test_labels), epochs=30, verbose=2,callbacks=[es]) 

Answer: Early Stopping Callback will search for a value that stopped increasing (or decreasing) so it's not a good use for your problem. However tf.keras allows you to use custom callbacks. For your example: class MyThresholdCallback(tf.keras.callbacks.Callback): def __init__(self, threshold): super(MyThresholdCallback, self).__init__() self.threshold = threshold def on_epoch_end(self, epoch, logs=None): val_acc = logs["val_acc"] if val_acc >= self.threshold: self.model.stop_training = True  For TF version 2.3 or above, you might have to use "val_accuracy" instead of "val_acc". Thank you Christian Westbrook for the note in the comments. The above Callback, on each epoch end, will extract Validation Accuracy from all available logs. Then it will compare it with user defined threshold (in your case 90%). If the criterion is met the training will be stopped. With that you can simply call: my_callback = MyThresholdCallback(threshold=0.9) history = model.fit(training_images, training_labels, validation_data=(test_images, test_labels), epochs=30, verbose=2, callbacks=[my_callback])  Alternatively, you can use def on_batch_end(...) if you want to stop immediately. This however, requires parameters batch, logs instead of epoch, logs.
Question: [in]: from sklearn_extra.cluster import KMedoids  [out]: ModuleNotFoundError: No module named 'sklearn_extra'  Then, I tried installing sklearn_extra via [in]: python -m pip install sklearn_extra  [out]: ERROR: Could not find a version that satisfies the requirement sklearn_extra (from versions: none) ERROR: No matching distribution found for sklearn_extra  Then, I went to installation part of the website(https://scikit-learn-extra.readthedocs.io/en/latest/install.html) and did what it said: Installation Dependencies scikit-learn-extra requires, Python (>=3.5) scikit-learn (>=0.21), and its dependencies Cython (>0.28) User installation Latest development version can be installed with, pip install https://github.com/scikit-learn-contrib/scikit-learn-extra/archive/master.zip  [in]: pip install https://github.com/scikit-learn-contrib/scikit-learn-extra/archive/master.zip  [out]:  ERROR: Command errored out with exit status 1: command: 'c:\users\m\appdata\local\programs\python\python37\python.exe' 'c:\users\m\appdata\local\programs\python\python37\lib\site-packages\pip' install --ignore-installed --no-user --prefix 'C:\Users\m\AppData\Local\Temp\pip-build-env-yopprv13\overlay' --no-warn-script-location --no-binary :none: --only-binary :none: -i https://pypi.org/simple -- setuptools wheel 'cython>=0.28' numpy==1.14.5 cwd: None Complete output (14 lines): Traceback (most recent call last): File "c:\users\m\appdata\local\programs\python\python37\lib\runpy.py", line 193, in _run_module_as_main "__main__", mod_spec) File "c:\users\m\appdata\local\programs\python\python37\lib\runpy.py", line 85, in _run_code exec(code, run_globals) File "c:\users\m\appdata\local\programs\python\python37\lib\site-packages\pip\__main__.py", line 16, in <module> from pip._internal.main import main as _main # isort:skip # noqa File "c:\users\m\appdata\local\programs\python\python37\lib\site-packages\pip\_internal\main.py", line 8, in <module> import locale File "c:\users\m\appdata\local\programs\python\python37\lib\locale.py", line 16, in <module> import re File "c:\users\m\appdata\local\programs\python\python37\lib\re.py", line 143, in <module> class RegexFlag(enum.IntFlag): AttributeError: module 'enum' has no attribute 'IntFlag'  I checked versions of Cython,Python and sklearn they satisfy the required range. Edit: the solution is to uninstall enum34 for me thanks to Balraj Ashwatt's comment. pip uninstall -y enum34  Then I was able to install sklearn_extra

Answer: Uninstalling enum34 worked for me and then I was able to install sklearn_extra pip uninstall -y enum34
Question: I've generated a list of combinations, using itertools and I'm getting a result that looks like this: nums = [-5,5,4,-3,0,0,4,-2] x = [x for x in set(itertools.combinations(nums, 4)) if sum(x)==target] >>> x = [(-5, 5, 0, 4), (-5, 5, 4, 0), (5, 4, -3, -2), (5, -3, 4, -2)]  What is the most time-complexity wise efficient way of removing unordered duplicates, such as x[0] and x[1] are the duplicates. Is there anything built in to handle this? My general approach would be to create a counter of all elements in one and compare to the next. Would this be the best approach? Thank you for any guidance.

Answer: Since you want to find unordered duplicates the best way to go is by typecasting. Typecast them as set. Since set only contains immutable elements. So, I made a set of tuples.  Note: The best way to eliminate duplicates is by making a set of the given elements.  >>> set(map(tuple,map(sorted,x))) {(-3, -2, 4, 5), (-5, 0, 4, 5)}
Question: I have a shell.nix that I use for Python development that looks like this:  with import <nixpkgs> {}; (( python37.withPackages (ps: with ps; [ matplotlib spacy pandas spacy_models.en_core_web_lg plotly ])).override({ignoreCollisions=true; })).env  It works fine for these packages. The problem is, I also want to use colormath, which doesn't seem to be in nixpkgs. How can I import that package?  I can generate a requirements.nix with pypi2nix -V python3 -e colormath, and I've tried to import it with something like this:  with import <nixpkgs> {}; let colormath = import ./requirements.nix { inherit pkgs; } in (( python37.withPackages (ps: with ps; [ matplotlib spacy ... colormath ])).override({ignoreCollisions=true; })).env  Edit: here's a gist of the output of requirements.nix.  I've also tried to make a python package nix expression, as in Nixpkgs, and it seems to build OK:  { buildPythonPackage , fetchPypi , networkx , numpy , lib , pytest }: buildPythonPackage rec { pname = "colormath"; version = "3.0.0"; src = fetchPypi { inherit version; inherit pname; sha256 = "3d4605af344527da0e4f9f504fad7ddbebda35322c566a6c72e28edb1ff31217"; }; checkInputs = [ pytest ]; checkPhase = '' pytest ''; # Tests seem to hang # doCheck = false; propagatedBuildInputs = [ networkx numpy ]; meta = { homepage = "https://github.com/gtaylor/python-colormath"; license = lib.licenses.bsd2; description = "Color math and conversion library."; }; }  (I even made a pull request for it.) But I just don't know how to import this into my development environment.  I'm still off. Is there an easy way to combine nixpkgs and non-nixpkgs python modules? 

Answer: I did solve a similar problem like that: with import <nixpkgs> {}; ( let colormath = pkgs.python37Packages.buildPythonPackage rec { pname = "colormath"; version = "3.0.0"; src = pkgs.python37Packages.fetchPypi{ inherit version; inherit pname; sha256 = "05qjycgxp3p2f9n6lmic68sxmsyvgnnlyl4z9w7dl9s56jphaiix"; }; buildInputs = [ pkgs.python37Packages.numpy pkgs.python37Packages.networkx ]; }; in pkgs.python37.buildEnv.override rec { extraLibs = [ pkgs.python37 pkgs.python37Packages.matplotlib pkgs.python37Packages.spacy pkgs.python37Packages.pandas pkgs.python37Packages.spacy_models.en_core_web_lg pkgs.python37Packages.plotly colormath ]; } ).env  There is probably room for improvements but this worked for me.
Question: I have a problem with passing the host's (Centos7) locales to the python3 docker image. Only the following locales end up in the image, even though I used the suggestion described in the link below: C C.UTF-8 POSIX  Why does locale.getpreferredencoding() return 'ANSI_X3.4-1968' instead of 'UTF-8'? My Dockerfile has: FROM python:3.7.5 ENV LC_ALL C.UTF-8 WORKDIR /data ADD ./requirements.txt /data/requirements.txt RUN pip install -r requirements.txt COPY . /data CMD [ "python3", "./test.py" ]  When I run this command: locale.setlocale(locale.LC_ALL,'ru_RU')  it throws this error: Traceback (most recent call last): File "./test.py", line 10, in <module> locale.setlocale(locale.LC_ALL,'ru_RU') File "/usr/local/lib/python3.7/locale.py", line 608, in setlocale return _setlocale(category, locale) locale.Error: unsupported locale setting  If I set  ENV LANG ru_RU.UTF-8 ENV LC_ALL ru_RU.UTF-8  Then I get: locale: Cannot set LC_CTYPE to default locale: No such file or directory locale: Cannot set LC_MESSAGES to default locale: No such file or directory locale: Cannot set LC_COLLATE to default locale: No such file or directory locale.getdefaultlocale ('ru_RU', 'UTF-8') locale.getpreferredencoding UTF-8 Exception: unsupported locale setting  Please, explain how can I add a ru_RU locale into the python image?

Answer: What I would do for Debian based docker image: FROM python:3.7.5 RUN apt-get update && \ apt-get install -y locales && \ sed -i -e 's/# ru_RU.UTF-8 UTF-8/ru_RU.UTF-8 UTF-8/' /etc/locale.gen && \ dpkg-reconfigure --frontend=noninteractive locales ENV LANG ru_RU.UTF-8 ENV LC_ALL ru_RU.UTF-8  and then in python: import locale locale.setlocale(locale.LC_ALL,'ru_RU.UTF-8')
Question: I'd like to plot several data into a grid using holoviews/hvplot, based on one dimension, which contains several unique data points. Considering this example: import seaborn as sns import hvplot.pandas iris = sns.load_dataset('iris') plot = iris.hvplot.scatter(x="sepal_length", y="sepal_width", col="species") hvplot.show(plot)  The above code creates several plots, based on the species part of the iris data set, resulting in the picture below:   But now imagine there were not 3 different species, but twenty. The plot would get to wide so I'd like to break the line after a few plots. But I couldn't find any "maximum columns" parameter. A normal grid expects another column to define the rows which I don't have. Any suggestions would help.

Answer: In your case I would not create a Gridspace (by using keyword 'row' and 'col') but a Layout.  When you have a Layout you can adjust the number of columns easily with .cols(2). Using hvplot you have to use keyword 'by' and 'subplots=True' instead of 'col'.  See the code below: iris.hvplot.scatter( x='sepal_length', y='sepal_width', by='species', subplots=True, ).cols(2)   Resulting plot: