eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I have a piece of pandas code which used to work in version 1.0.5. Here's a simplified, self-contained example of my problem: import pandas as pd df = pd.DataFrame(data=[ ('bk1', 10), ('bk1', None), ('bk1', 13), ('bk1', None), ('bk2', None), ('bk2', 14), ('bk3', 12), ('bk3', None), ], columns=('book', 'price')) grouped = df.groupby(['book'], as_index=False, sort=False) df = grouped.fillna(method='ffill') print(df)  In this example, we have a list of book sales, where some of the prices are missing. We're trying to fill in the missing data by using the previous row, where that row is the same book. In Pandas 1.0.5, this produces a dataframe with two columns:  book price 0 bk1 10.0 1 bk1 10.0 2 bk1 13.0 3 bk1 13.0 4 bk2 NaN 5 bk2 14.0 6 bk3 12.0 7 bk3 12.0  In Pandas 1.1.0, this removes the book column, which makes the output unusable.  price 0 10.0 1 10.0 2 13.0 3 13.0 4 NaN 5 14.0 6 12.0 7 12.0  I have read the patch notes for version 1.1.0, and I can't find any remarks about this change. Questions:  Is this a bug in Pandas, or am I relying on undefined behavior? Is there a more natural way to express this?  Questions you might ask:  Why not use fillna without a groupby? In this example, the first row with bk2 has no price, but it wouldn't make any sense to fill it in with the previous row, which is the price of bk1.  Why use ffill instead of dropping NA values? My real code is working with timeseries data, and ffill is the most natural way to express carrying forward the last known observation.  

Answer: You could take a different approach to get around this issue (different from the solution proposed by Nick ODell) by using the update function: df.update(df.groupby(['book']).ffill()) print(df) Out[1]: book price 0 bk1 10.0 1 bk1 10.0 2 bk1 13.0 3 bk1 13.0 4 bk2 NaN 5 bk2 14.0 6 bk3 12.0 7 bk3 12.0  This also works in both versions.
Question: I am trying to install mysqlclient for python, but I always get this error when I try: $ pip3 install mysqlclient   How can I resolve this issue?

Answer: Looking at the error message, it seems that there is an OS Error, and judging by the terminal layout, you're using Linux. It seems that to install this package on Linux, there are extra instructions to follow, mentioned on the module's github page:  You may need to install the Python 3 and MySQL development headers and libraries like so:  $ sudo apt-get install python3-dev default-libmysqlclient-dev build-essential # Debian / Ubuntu % sudo yum install python3-devel mysql-devel # Red Hat / CentOS  Then you can install mysqlclient via pip now: $ pip install mysqlclient
Question: I keep on getting this error related to input shape. Any help would be highly appreciated. Thanks! import tensorflow as tf (xtrain, ytrain), (xtest, ytest) = tf.keras.datasets.mnist.load_data() model = tf.keras.Sequential([ tf.keras.layers.Conv2D(16, kernel_size=3, activation='relu'), tf.keras.layers.MaxPooling2D(pool_size=2), tf.keras.layers.Conv2D(32, kernel_size=3, activation='relu'), tf.keras.layers.MaxPooling2D(pool_size=2), tf.keras.layers.Flatten(), tf.keras.layers.Dense(64, activation='relu'), tf.keras.layers.Dense(10, activation='softmax') ]) model.compile(loss='categorical_crossentropy', optimizer='adam', metrics='accuracy') history = model.fit(xtrain, ytrain, validation_data=(xtest, ytest), epochs=10, batch_size=8)   ValueError: Input 0 of layer sequential is incompatible with the layer: : expected min_ndim=4, found ndim=3. Full shape received: [8, 28, 28] 

Answer: The input layers of the model you created needs a 4 dimension tensor to work with but the x_train tensor you are passing to it has only 3 dimensions This means that you have to reshape your training set with .reshape(n_images, 286, 384, 1). Now you have added an extra dimension without changing the data and your model is ready to run. you need to reshape your x_train tensor to a 4 dimension before training your model. for example: x_train = x_train.reshape(-1, 28, 28, 1)  for more info on keras inputs Check this answer
Question: I'm trying to install jupyter on Ubuntu 16.04.6 x64 on DigitalOcean droplet. It is giving me the following error message, and I can't understand what this means.  ERROR: After October 2020 you may experience errors when installing or updating packages. This is because pip will change the way that it resolves dependency conflicts. We recommend you use --use-feature=2020-resolver to test your packages with the new resolver before it becomes the default. jsonschema 3.2.0 requires six>=1.11.0, but you'll have six 1.10.0 which is incompatible  Any help would be greatly appreciated!

Answer: According to this announcement, pip will introduce a new dependency resolver in October 2020, which will be more robust but might break some existing setups. Therefore they are suggesting users to try running their pip install scripts at least once (in dev mode) with this option: --use-feature=2020-resolver to anticipate any potential issue before the new resolver becomes the default in October 2020 with pip version 20.3.  On behalf of the PyPA, I am pleased to announce that we have just released pip 20.2, a new version of pip. You can install it by running python -m pip install --upgrade pip. The highlights for this release are:  The beta of the next-generation dependency resolver is available Faster installations from wheel files Improved handling of wheels containing non-ASCII file contents Faster pip list using parallelized network operations Installed packages now contain metadata about whether they were directly requested by the user (PEP 376’s REQUESTED file)  The new dependency resolver is off by default because it is not yet ready for everyday use. The new dependency resolver is significantly stricter and more consistent when it receives incompatible instructions, and reduces support for certain kinds of constraints files, so some workarounds and workflows may break. Please test it with the --use-feature=2020-resolver flag. Please see our guide on how to test and migrate, and how to report issues . We are preparing to change the default dependency resolution behavior and make the new resolver the default in pip 20.3 (in October 2020).
Question: I'm trying to add a pyproject.toml to a project that's been using setup.py in order to enable support by pipx. I'd like to specify the command line scripts the project includes in pyproject.toml, but all the guides I can find give instructions for use with poetry, which I am not using. I also don't want to specify entry points to modules - I already have working command line scripts and just want to specify those. Is there a proper place in pyproject.toml to specify command line scripts? Not sure it matters, but the package in question is cutlet.

Answer:  Is there a proper place in pyproject.toml to specify command line scripts?  PEP566 (Metadata 2.1) only defines Core metadata specifications. Thus, the answer depends on your build system (Note: PEP518 defines build system concept). If you use the existing build tools such as setuptools, poetry, and flit, you only can consider adding such options in pyproject.toml if that tool supports command line scripts (console_scripts) in pyproject.toml. Surely, if you have your own build tool, you need to implement a parser to parse the command line scripts in pyproject.toml. Lastly, you can check below list to know which major build system supports command line scripts (console_scripts) in pyproject.toml (2020/Oct):  setuptools: not implemented yet according to PEP621 poetry: yes, here's part of the implementation. flit: yes.
Question: It's awesome how google something can be annoying when you can't find the right words. I found a million answers on how about to create a Telegram Bot to send and receive messages, and it's easy as write maybe five code lines. But how about managing my own account? I want to know if it it's posible, using Python (telepot or other library), to retrieve my personal messages and send messages from my PERSONAL account, not using a bot. If it's possible, where can I find more information about that

Answer: Telegram has a thorough and documented public API. Following some links from there, here is the summary of the relevant parts:  the API is not restricted to bots, they are just a (special) kind of users; the API has methods called getMessages and sendMessage, that should be what you need; to call the API, Telegram recommends to use the dedicated library TDLib available for multiple programming languages. There are several examples available on GitHub   Among the examples, if you go the the Python part, they recommend:  If you use modern Python >= 3.6, take a look at python-telegram.  You'll find instructions to use the library, and in the examples folder you can find a script to send a message. I'll copy it here for the sake of completeness: import logging import argparse from utils import setup_logging from telegram.client import Telegram """ Sends a message to a chat Usage: python examples/send_message.py api_id api_hash phone chat_id text """ if __name__ == '__main__': setup_logging(level=logging.INFO) parser = argparse.ArgumentParser() parser.add_argument('api_id', help='API id') # https://my.telegram.org/apps parser.add_argument('api_hash', help='API hash') parser.add_argument('phone', help='Phone') parser.add_argument('chat_id', help='Chat id', type=int) parser.add_argument('text', help='Message text') args = parser.parse_args() tg = Telegram( api_id=args.api_id, api_hash=args.api_hash, phone=args.phone, database_encryption_key='changeme1234', ) # you must call login method before others tg.login() # if this is the first run, library needs to preload all chats # otherwise the message will not be sent result = tg.get_chats() # `tdlib` is asynchronous, so `python-telegram` always returns you an `AsyncResult` object. # You can wait for a result with the blocking `wait` method. result.wait() if result.error: print(f'get chats error: {result.error_info}') else: print(f'chats: {result.update}') result = tg.send_message( chat_id=args.chat_id, text=args.text, ) result.wait() if result.error: print(f'send message error: {result.error_info}') else: print(f'message has been sent: {result.update}')  Of course you'll need to explore the documentation to get what are all those variables / ids in your case, but it will get you started!
Question: I am running pipenv install --dev which is giving me the following errors Courtesy Notice: Pipenv found itself running within a virtual environment, so it will automatically use that environment, instead of creating its own for any project. You can set PIPENV_IGNORE_VIRTUALENVS=1 to force pipenv to ignore that environment and create its own instead. You can set PIPENV_VERBOSITY=-1 to suppress this warning. Installing dependencies from Pipfile.lock (2df4c1)… Failed to load paths: /bin/sh: /Users/XXXX/.local/share/virtualenvs/my-service-enGYxXYk/bin/python: No such file or directory Output: Failed to load paths: /bin/sh: /Users/XXXX/.local/share/virtualenvs/my-service-enGYxXYk/bin/python: No such file or directory Output: Failed to load paths: /bin/sh: /Users/XXXX/.local/share/virtualenvs/my-service-enGYxXYk/bin/python: No such file or directory  I don't really want to change the command around I would rather solve the underlying issue as it is part of a package.json file in a project others are using rather than something i am just trying to run on my own machine.. Thanks

Answer: Remove your Pipfile.lock and try rerunning pipenv install to rebuild your dependencies from your Pipfile. It is looking for a virtual environment that does not exist. By removing your Pipfile.lock, you force pipenv to create a new environment.
Question: I want to get my Python script working on a bare metal device like microcontroller WITHOUT the need for an interpreter. I know there are already JIT compilers for Python like PyPy, and interpreters like CPython. However, existing interpreters I've seen (such as CPython) take up large memory (in MB range). Is there an AOT compiler for Python (i.e. compiling directly to native hardware through intermediary like LLVM)? I assume such a compiler would enable Python to run much faster compared to existing implementations AND with lower memory footprint. If there is, I wonder why that solution hasn't been popularized.

Answer: As you already mentioned Cython is an option (However, it is true that the result is big due since the C runtime need to implement the Python functionality together with your program). With regards to LLVM there was a project by Google named unladen swallow. However, that project is mostly abandoned. You can find some information about it here Basically it was an attempt to bring LLVM optimizations into the runtime of Cython. E.g JITTING Python code. Another old alternative was shed skin which compiles Python to C++. Some information about it can be found here. Yet another option similar to shed skin is to restrict yourself to a subset of the Python language and use micropython.
Question: I want to use argeparse module in following way, from argparse import ArgumentParser parser = ArgumentParser() parser.add_argument('-b', dest='binKey', type=str) args = parser.parse_args() # I will make use of args.binKey option in this space print args.binKey # After that I want to add -d option to the arguments parser.add_argument('-d', dest='dir', type=str) args = parser.parse_args()  With this example, I can only provide -b option as shown below in the help text. $$ python test.py -h usage: test.py [-h] [-b BINKEY] optional arguments: -h, --help show this help message and exit -b BINKEY  What I want is I should be able to provide both options while running this code and those options should also by visible in --help option NOTE: I know that I can provide both options before executing parse_args() once, but that is not the way I want to use my parser.

Answer: With the help from @hpaulj's comment, here is the resolution:  Turn off help option while creating ArgumentParser object. You can call parse_known_args multiple times in between. Add help option at the end.  Below is a snippet of the code from argparse import ArgumentParser parser = ArgumentParser(add_help=False) parser.add_argument('-b', dest='binKey', type=str) args = parser.parse_known_args()[0] # Here you can make use of args.binKey option in this space print args.binKey # You can more options after this as parser.add_argument('-d', dest='dir', type=str) parser.add_argument('-h', '--help', action='help', default='==SUPPRESS==', help=('show this help message and exit')) args = parser.parse_args()
Question: I have a Cloud Function in GCP that queries BigQuery in a specific project/environment. As I have multiple environments I would like to get the current project/environment of the cloud function. This is so that I can access BigQuery in the corresponding environment. Of course I could just hardcode the project_id, but I would like to do this programmatically. According to Google environment variables are set automatically. But when I try to access those I cannot find any of them. For instance I have tried the following, which gives me none of the env. var listed by Google. print(os.environ)  Anyone managed to access environment variables at runtime?

Answer: Those environment variables you are referring to only applies to python 3.7, the second section on that page (https://cloud.google.com/functions/docs/env-var#nodejs_10_and_subsequent_runtimes) states :  All languages and runtimes other than those mentioned in the previous section will use this more limited set of predefined environment variables.  This means the GCP_PROJECT is no longer set with 3.8 (at least for now).
Question: I noticed one strange thing that autopep8 autoformatting in VSCode doesn't work when we set  "python.formatting.autopep8Args": [ "--line-length 119" ],  But if this setting is in a default mode that is line length 79 then it works well. Is there some issue with autopep8 to work only with line length 79 not more than that, or I am making any mistake in VSCode. Major feature that I need is when my python program line goes too long it should be able to break it in multiple lines. I don't want to go ahead with 79 characters' approach. My preferred approach is 119 characters. Currently, I have to indent big lines manually. Is there any other format apart from pep8 which supports 119 characters and indent lines with characters more than 119 characters automatically. I am attaching my settings.json file data { "window.zoomLevel": 1, "python.dataScience.sendSelectionToInteractiveWindow": true, "diffEditor.ignoreTrimWhitespace": false, "editor.fontSize": 16, "python.formatting.provider": "autopep8", "editor.formatOnPaste": true, "editor.formatOnSave": true, "editor.formatOnType": true, "python.autoComplete.addBrackets": true, "python.formatting.autopep8Args": [ "--line-length 119" ], // "python.linting.flake8Args": [ // "--max-line-length=120" // ], "files.autoSaveDelay": 10000, "editor.defaultFormatter": "ms-python.python", "files.autoSave": "afterDelay", "files.trimTrailingWhitespace": true, "files.trimFinalNewlines": true, "editor.quickSuggestions": true, "editor.codeActionsOnSave": null, } 

Answer: experimental worked for me "python.formatting.autopep8Args": ["--max-line-length", "120", "--experimental"]  check out this link for proper format specifier settings
Question: We have 100+ private packages and so far we've been using s3pypi to set up a private pypi in an s3 bucket. Our private packages have dependencies on each other (and on public packages), and it is (of course) important that our GitLab pipelines find the latest functional version of packages it relies on. I.e. we're not interested in the latest checked in code. We create new wheels only after tests and qa has run against a push to master (which is a long-winded way of explaining that -e <vcs> requirements will not work). Our setup works really well until someone creates a new public package on the official pypi that shadows one of our package names. We can force our private package to be chosen by increasing the version number so it is higher than the new package on pypi.org - or by renaming our package to something that haven't yet been taken on pypi.org. This is obviously a hacky and fragile solution, but apparently the functionality is this way by-design. After the initial bucket setup s3pypi has required no maintenance or administration. The above ticket suggests using devpi but that seems like a very heavy solution that requires administration/monitoring/etc. GitLab's pypi solution seems to be at individual package level (meaning we'd have to list up to 100+ urls - one for each package). This doesn't seem practical, but maybe I'm not understanding something (I can see the package registry menu under our group as well, but the docs point to the "package-pypi" docs). We can't be the first small company that has faced this issue..? Is there a better way than to register dummy versions of all our packages on pypi.org (with version=0.0.1, so the s3pypi version will be preferred)?

Answer: It might not be the solution for you, but I tell what we do.  Prefix the package names, and using namespaces (eg. company.product.tool). When we install our packages (including their in-house dependencies), we use a requirements.txt file including our PyPI URL. We run everything in container(s) and we install all public dependencies in them when we are building the images.
Question: I have a two time series in separate pandas.dataframe, the first one - series1 has less entries and different start datatime from the second - series2: index1 = pd.date_range(start='2020-06-16 23:16:00', end='2020-06-16 23:40:30', freq='1T') series1 = pd.Series(range(len(index1)), index=index1) index2 = pd.date_range('2020-06-16 23:15:00', end='2020-06-16 23:50:30', freq='30S') series2 = pd.Series(range(len(index2)), index=index2)  How can I resample series2 to match the DatetimeIndex of series1?

Answer: Use reindex: series2.reindex(series1.index)  Output: 2020-06-16 23:16:00 2 2020-06-16 23:17:00 4 2020-06-16 23:18:00 6 2020-06-16 23:19:00 8 2020-06-16 23:20:00 10 2020-06-16 23:21:00 12 2020-06-16 23:22:00 14 2020-06-16 23:23:00 16 2020-06-16 23:24:00 18 2020-06-16 23:25:00 20 2020-06-16 23:26:00 22 2020-06-16 23:27:00 24 2020-06-16 23:28:00 26 2020-06-16 23:29:00 28 2020-06-16 23:30:00 30 2020-06-16 23:31:00 32 2020-06-16 23:32:00 34 2020-06-16 23:33:00 36 2020-06-16 23:34:00 38 2020-06-16 23:35:00 40 2020-06-16 23:36:00 42 2020-06-16 23:37:00 44 2020-06-16 23:38:00 46 2020-06-16 23:39:00 48 2020-06-16 23:40:00 50 Freq: T, dtype: int64
Question: When I tried to use virtualenv on Ubuntu 18.04, I got this error:  bash: /usr/local/bin/virtualenv: /usr/bin/python: bad interpreter: No such file or directory  Python 2 and 3 is working fine: josir@desenv16:~/bin$ which python3 /usr/bin/python3 josir@desenv16:~/bin$ python3 Python 3.6.9 (default, Apr 18 2020, 01:56:04) [GCC 8.4.0] on linux  I've already tried to unistall virtualenv: sudo apt-get purge --auto-remove virtualenv sudo apt-get purge --auto-remove python-virtualenv sudo apt-get purge --auto-remove python3-virtualenv  But when I installed again, the error remains.

Answer:  bash: /usr/local/bin/virtualenv: /usr/bin/python: bad interpreter: No such file or directory  The error is in '/usr/local/bin/virtualenv' — it's first line (shebang) is #!/usr/bin/python and there is no such file at your system. I believe the stream of events led to the situation is: you've installed virtualenv with pip (not apt) long ago and put /usr/local/bin at the front of your $PATH. Then you upgraded you system; the upgrade removed /usr/bin/python, now you have only /usr/bin/python3. Now you have to decide which route you'd go: apt or pip. If you choose apt — remove /usr/local/bin/virtualenv. If you choose pip: my advice is to uninstall as much as possible python packages installed with apt; reinstall virtualenv; that should be the only additional package installed with apt. For every project/task create a virtual environment and install packages with pip. PS. Personal experience: I switched from apt way to pip a few years ago. PPS. Avoid using sudo pip — do not clobber system installation. Either install into virtual environments or pip install --user.
Question: I need to create a very large numpy array that will hold non-negative integer values. I know in advance what the largest integer will be, so I want to try to use the smallest datatype possible. So far I have the following: >>> import numpy as np >>> def minimal_type(max_val, types=[np.uint8,np.uint16,np.uint32,np.uint64]): ''' finds the minimal data type needed to correctly store the given max_val returns None if none of the provided types are sufficient ''' for t in types: if max_val <= np.iinfo(t).max: return t return None >>> print(minimal_type(42)) <class 'numpy.uint8'> >>> print(minimal_type(255)) <class 'numpy.uint8'> >>> print(minimal_type(256)) <class 'numpy.uint16'> >>> print(minimal_type(4200000000)) <class 'numpy.uint32'> >>>  Is there a numpy builtin way to achieve this functionality?

Answer: It's numpy.min_scalar_type. Examples from the docs: >>> np.min_scalar_type(10) dtype('uint8') >>> np.min_scalar_type(-260) dtype('int16') >>> np.min_scalar_type(3.1) dtype('float16') >>> np.min_scalar_type(1e50) dtype('float64') >>> np.min_scalar_type(np.arange(4,dtype='f8')) dtype('float64')  You might not be interested in the behavior for floats, but I'm including it anyway for other people who come across the question, particularly since the use of float16 and the lack of float->int demotion might be surprising.