eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: Hey I know there is a solution for this in Java, I'm curious to know if anyone knows of a Python 3 solution for converting a JSON object or file into protobuf format. I would accept either or as converting to an object is trivial. Searching the stackoverflow site, I only found examples of protobuf->json, but not the other way around. There is one extremely old repo that may do this but it is in Python 2 and our pipeline is Python 3. Any help is as always, appreciated.

Answer: The library you're looking for is google.protobuf.json_format. You can install it with the directions in the README here. The library is compatible with Python >= 2.7. Example usage: Given a protobuf message like this: message Thing { string first = 1; bool second = 2; int32 third = 3; }  You can go from Python dict or JSON string to protobuf like: import json from google.protobuf.json_format import Parse, ParseDict d = { "first": "a string", "second": True, "third": 123456789 } message = ParseDict(d, Thing()) # or message = Parse(json.dumps(d), Thing()) print(message.first) # "a string" print(message.second) # True print(message.third) # 123456789  or from protobuf to Python dict or JSON string: from google.protobuf.json_format import MessageToDict, MessageToJson message_as_dict = MessageToDict(message) message_as_dict['first'] # == 'a string' message_as_dict['second'] # == True message_as_dict['third'] # == 123456789 # or message_as_json_str = MessageToJson(message)  The documentation for the json_format module is here.
Question: I am trying to normally import the TensorFlow python package, but I get the following error:  Here is the text from the above terminal image: 2020-02-23 19:01:06.163940: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libnvinfer.so.6'; dlerror: libnvinfer.so.6: cannot open shared object file: No such file or directory 2020-02-23 19:01:06.164019: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libnvinfer_plugin.so.6'; dlerror: libnvinfer_plugin.so.6: cannot open shared object file: No such file or directory 2020-02-23 19:01:06.164030: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:30] Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly. <module 'tensorflow_core._api.v2.version' from '/home/saman/miniconda3/envs/testconda/lib/python3.7/site-packages/tensorflow_core/_api/v2/version/__init__.py' 

Answer: This is a warning, not an error. You can still use TensorFlow. The shared libraries libnvinfer and libnvinfer_plugin are optional and required only if you are using nvidia's TensorRT capabilities. To suppress this and all other warnings, set the environment variable TF_CPP_MIN_LOG_LEVEL="2".
Question: I am so new on Image Processing and what I'm trying to do is clearing the noise from captchas; For captchas, I have different types of them:    For the first one what I did is :    Firstly, I converted every pixel that is not black to the black. Then, I found a pattern that is a noise from the image and deleted it. For the first captcha, it was easy to clear it and I found the text with tesseract. But I am looking for a solution for the second and the third. How this must go like? I mean what are the possible methods to clear it? This is how I delete patterns: def delete(searcher,h2,w2): h = h2 w = w2 search = searcher search = search.convert("RGBA") herear = np.asarray(search) bigar = np.asarray(imgCropped) hereary, herearx = herear.shape[:2] bigary, bigarx = bigar.shape[:2] stopx = bigarx - herearx + 1 stopy = bigary - hereary + 1 pix = imgCropped.load() for x in range(0, stopx): for y in range(0, stopy): x2 = x + herearx y2 = y + hereary pic = bigar[y:y2, x:x2] test = (pic == herear) if test.all(): for q in range(h): for k in range(w): pix[x+k,y+q] = (255,255,255,255)  Sorry for the variable names, I was just testing function. Thanks..

Answer: Here is my solution,  Firstly I got the background pattern(Edited on paint by hand). From:  After that, I created a blank image to fill it with differences between the pattern and image. img = Image.open("x.png").convert("RGBA") pattern = Image.open("y.png").convert("RGBA") pixels = img.load() pixelsPattern = pattern.load() new = Image.new("RGBA", (150, 50)) pixelNew = new.load() for i in range(img.size[0]): for j in range(img.size[1]): if(pixels[i,j] != pixelsPattern[i,j]): pixelNew[i,j] = pixels[i,j] new.save("differences.png")  Here are the differences..  and finally, I added blur and cleared the bits which are not black. Result :  With pytesseract result is 2041, it is wrong for this image but the general rate is around %60.
Question: I was wondering what's the difference between these two cases? Is the inner structure of the modules somehow different? So why this one works: >>> import numpy >>> numpy.random.RandomState <class 'numpy.random.mtrand.RandomState'>  But this one doesn't work until I import the nested module also: >>> import tkinter >>> tkinter.ttk.Spinbox Traceback (most recent call last): File "<stdin>", line 1, in <module> AttributeError: module 'tkinter' has no attribute 'ttk' >>> import tkinter.ttk >>> tkinter.ttk.Spinbox <class 'tkinter.ttk.Spinbox'>  I think that this must be something to do with the __init__.py files in each module, but a concrete example of an implementation would be helpful.

Answer: This kind of import behavior can be achieved by importing the deep class in the respective __init__.py files. Here's a quick demonstration of a project structure that works pretty much exactly like the real numpy and tkinter packages did: . ├── main.py ├── numpy │ ├── __init__.py │ └── random │ ├── __init__.py │ └── mtrand.py └── tkinter ├── __init__.py └── ttk.py  numpy/__init__.py: from .random import RandomState  numpy/random/__init__.py: from .mtrand import RandomState  numpy/random/mtrand.py: class RandomState: pass  tkinter/__init__.py:  tkinter/ttk.py: class Spinbox: pass  main.py: import numpy import tkinter print(numpy.random.RandomState) try: print(tkinter.ttk.Spinbox) except AttributeError as e: print("Error:", e) import tkinter.ttk print(tkinter.ttk.Spinbox)   Output: <class 'numpy.random.mtrand.RandomState'> Error: module 'tkinter' has no attribute 'ttk' <class 'tkinter.ttk.Spinbox'>
Question: I am trying to convert the shape property of a Tensor in Tensorflow 2.1 and I get this error: AttributeError: 'Tensor' object has no attribute 'numpy'  I already checked that the output of tf.executing eagerly() is True, A bit of context: I load a tf.data.Dataset from a TFRecords, then I apply a map. The maping function is trying to convert the shape property of one of the dataset sample Tensor to numpy: def _parse_and_decode(serialized_example): """ parse and decode each image """ features = tf.io.parse_single_example( serialized_example, features={ 'encoded_image': tf.io.FixedLenFeature([], tf.string), 'kp_flat': tf.io.VarLenFeature(tf.int64), 'kp_shape': tf.io.FixedLenFeature([3], tf.int64), } ) image = tf.io.decode_png(features['encoded_image'], dtype=tf.uint8) image = tf.cast(image, tf.float32) kp_shape = features['kp_shape'] kp_flat = tf.sparse.to_dense(features['kp_flat']) kp = tf.reshape(kp_flat, kp_shape) return image, kp def read_tfrecords(records_dir, batch_size=1): # Read dataset from tfrecords tfrecords_files = glob.glob(os.path.join(records_dir, '*')) dataset = tf.data.TFRecordDataset(tfrecords_files) dataset = dataset.map(_parse_and_decode, num_parallel_calls=batch_size) return dataset def transform(img, labels): img_shape = img.shape # type: <class 'tensorflow.python.framework.ops.Tensor'>` img_shape = img_shape.numpy() # <-- Throws the error # ... dataset = read_tfrecords(records_dir)  This throws the error: dataset.map(transform, num_parallel_calls=1)  While this perfecly works: for img, labels in dataset.take(1): print(img.shape.numpy())  Edit: trying to access the img.numpy() instead of img.shape.numpy() results in the same behavior in the tranformer and the codde just above. I checked the type of img_shape and it is <class 'tensorflow.python.framework.ops.Tensor'>. Has anyone solved this sort of issue in new versions of Tensorflow?

Answer: The problem in your code is that you cannot use .numpy() inside functions that are mapped onto tf.data.Datasets, because .numpy() is Python code not pure TensorFlow code. When you use a function like my_dataset.map(my_function), you can only use tf.* functions inside your my_function function. This is not a bug of TensorFlow 2.x versions, but rather on how static graphs are generated behind the scenes for performance purposes. If you want to use custom Python code inside a function which you map on your dataset, you have to use tf.py_function(), docs: https://www.tensorflow.org/api_docs/python/tf/py_function. There is really no other way to mix Python code and TensorFlow code when mapping on a dataset. You can also consult this question for further information; it's the exact question that I asked a couple of months ago: Is there an alternative to tf.py_function() for custom Python code?
Question: I have built a python script using tensorflow and I am now trying to convert it to an .exe file, but have ran into a problem. After using pyinstaller and running the program from the command prompt I get the following error:  File "site-packages\tensorflow_core\python\pywrap_tensorflow.py", line 25, in <module> ModuleNotFoundError: No module named 'tensorflow.python.platform'  I have tried --hidden-import tensorflow.python.platform but it seems to have fixed nothing. (The program runs just fine in the interpreter) Your help would be greatly appreciated.

Answer: EDIT: The latest versions of PyInstaller (4.0+) now include support for tensorflow out of the box. Create a directory structure like this: - main.py # Your code goes here - don't bother actually naming you file this - hooks - hook-tensorflow.py  Copy the following into hook-tensorflow.py: from PyInstaller.utils.hooks import collect_all def hook(hook_api): packages = [ 'tensorflow', 'tensorflow_core', 'astor' ] for package in packages: datas, binaries, hiddenimports = collect_all(package) hook_api.add_datas(datas) hook_api.add_binaries(binaries) hook_api.add_imports(*hiddenimports)  Then, when compiling, add the command line option --additional-hooks-dir=hooks. If you come across more not found errors, simply add the full import name into the packages list. PS - for me, main.py was simply from tensorflow import *
Question: I am trying to plot two different charts in python through plotly. I have two plots, one plot consists of merged graph ( line and bar chart) like the following, ,  and another one is bar chart as follows,  I wanted to display one single chart with these two combined charts and display the same. I have tried this in plotly through make_subplots but I am not able to achieve the results properly. Below are the codes for creating these two charts, Line_Bar_chart Code: import plotly.graph_objects as go from plotly.offline import iplot trace1 = go.Scatter( mode='lines+markers', x = df['Days'], y = df['Perc_Cases'], name="Percentage Cases", marker_color='crimson' ) trace2 = go.Bar( x = df['Days'], y = df['Count_Cases'], name="Absolute_cases", yaxis='y2', marker_color ='green', marker_line_width=1.5, marker_line_color='rgb(8,48,107)', opacity=0.5 ) data = [trace1, trace2] layout = go.Layout( title_text='States_Name', yaxis=dict( range = [0, 100], side = 'right' ), yaxis2=dict( overlaying='y', anchor='y3', ) ) fig = go.Figure(data=data, layout=layout) iplot(fig, filename='multiple-axes-double') **Line_Bar_chart Code**:  Bar_chart Code: trace2 = go.Bar( x = df['Days'], y = df['Perc_Cases'], yaxis='y2', marker_color ='green', marker_line_width=1.5, marker_line_color='rgb(8,48,107)', opacity=0.5, ) layout = go.Layout( title_text='States_Name', yaxis2=dict( overlaying='y', ) ) fig = go.Figure(data=trace2, layout=layout) iplot(fig, filename='multiple-axes-double')  Any help on how to make subplots of these two graphs like below would be helpful, 

Answer: The key here is to assign your traces to the subplot through row and col in fig.add_trace(). And you don't have to use from plotly.offline import iplot for the latest plotly updates. Plot:  Code: # imports from plotly.subplots import make_subplots import plotly.graph_objects as go import pandas as pd import numpy as np # data df = pd.DataFrame({'Index': {0: 1.0, 1: 2.0, 2: 3.0, 3: 4.0, 4: 5.0, 5: 6.0, 6: 7.0, 7: 8.0, 8: 9.0, 9: 10.0}, 'A': {0: 15.0, 1: 6.0, 2: 5.0, 3: 4.0, 4: 3.0, 5: 2.0, 6: 1.0, 7: 0.5, 8: 0.3, 9: 0.1}, 'B': {0: 1.0, 1: 4.0, 2: 2.0, 3: 5.0, 4: 4.0, 5: 6.0, 6: 7.0, 7: 2.0, 8: 8.0, 9: 1.0}, 'C': {0: 12.0, 1: 6.0, 2: 5.0, 3: 4.0, 4: 3.0, 5: 2.0, 6: 1.0, 7: 0.5, 8: 0.2, 9: 0.1}}) # set up plotly figure fig = make_subplots(1,2) # add first bar trace at row = 1, col = 1 fig.add_trace(go.Bar(x=df['Index'], y=df['A'], name='A', marker_color = 'green', opacity=0.4, marker_line_color='rgb(8,48,107)', marker_line_width=2), row = 1, col = 1) # add first scatter trace at row = 1, col = 1 fig.add_trace(go.Scatter(x=df['Index'], y=df['B'], line=dict(color='red'), name='B'), row = 1, col = 1) # add first bar trace at row = 1, col = 2 fig.add_trace(go.Bar(x=df['Index'], y=df['C'], name='C', marker_color = 'green', opacity=0.4, marker_line_color='rgb(8,48,107)', marker_line_width=2), row = 1, col = 2) fig.show()
Question: I have problem when trying to run a website in Django: OSError: no library called "libcairo-2" was found cannot load library 'libcairo.so.2': /lib/x86_64-linux-gnu/libfontconfig.so.1: undefined symbol: FT_Done_MM_Var cannot load library 'libcairo.so': /lib/x86_64-linux-gnu/libfontconfig.so.1: undefined symbol: FT_Done_MM_Var cannot load library 'libcairo.2.dylib': libcairo.2.dylib: cannot open shared object file: No such file or directory cannot load library 'libcairo-2.dll': libcairo-2.dll: cannot open shared object file: No such file or directory  Althought the package is installed.  I have installed weasyprint pip3 install weasyprint python -m pip install WeasyPrint sudo apt-get install build-essential python3-dev python3-pip python3-setuptools python3-wheel python3-cffi libcairo2 libpango-1.0-0 libpangocairo-1.0-0 libgdk-pixbuf2.0-0 libffi-dev shared-mime-info  I also tried sudo apt install libcairo2-dev sudo apt install libgirepository1.0-dev  I have Lubuntu system. Any ideas how can I fix it? Thanks in advance

Answer: For Ubuntu 20 and Debian based distro, try: sudo apt-get install libpangocairo-1.0-0
Question: I just got Black and Pre-Commit set up for my Django repository. I used the default config for Black from the tutorial I followed and it's been working great, but I am having trouble excluding my migrations files from it. Here is the default configuration I've been using: pyproject.toml [tool.black] line-length = 79 include = '\.pyi?$' exclude = ''' /( \.git | \.hg | \.mypy_cache | \.tox | \.venv | _build | buck-out | build | dist )/ '''  I used Regex101.com to make sure that ^.*\b(migrations)\b.*$ matched apps/examples/migrations/test.py. [tool.black] line-length = 79 include = '\.pyi?$' exclude = ''' /( \.git | \.hg | \.mypy_cache | \.tox | \.venv | _build | buck-out | build | dist | ^.*\b(migrations)\b.*$ )/ '''  When I add that regex line to my config file, and run pre-commit run --all-files, it ignores the .git folder but still formats the migrations files. 

Answer: Add the migration exclusion to your .pre-commit-config.yaml file - id: black exclude: ^.*\b(migrations)\b.*$
Question: I was reading the BERT paper and was not clear regarding the inputs to the transformer encoder and decoder.  For learning masked language model (Cloze task), the paper says that 15% of the tokens are masked and the network is trained to predict the masked tokens. Since this is the case, what are the inputs to the transformer encoder and decoder?  Is the input to the transformer encoder this input representation (see image above). If so, what is the decoder input? Further, how is the output loss computed? Is it a softmax for only the masked locations? For this, the same linear layer is used for all masked tokens?

Answer: Ah, but you see, BERT does not include a Transformer decoder. It is only the encoder part, with a classifier added on top. For masked word prediction, the classifier acts as a decoder of sorts, trying to reconstruct the true identities of the masked words. Classifying Non-masked is not included in the classification task and does not effect loss. BERT is also trained on predicting whether a pair of sentences really does precedes one another or not. I do not remember how the two losses are weighted. I hope this draws a clearer picture.
Question: I'm using flask and have a lot of requests. The json module, which is used by flask, is quite slow. I automatically can use simplejson, but thats a bit slower, not faster. According to the documentation I can define a decoder (flask.json_decoder), but orjson doesn't have this class. I only have the function loads and dumps. Can somebody explain me, how I can exchange the json module with orjson? In the end I just want to use the loads and dumps function, but I can't connect my loose ends.

Answer: a very basic implementation could look like this: class ORJSONDecoder: def __init__(self, **kwargs): # eventually take into consideration when deserializing self.options = kwargs def decode(self, obj): return orjson.loads(obj) class ORJSONEncoder: def __init__(self, **kwargs): # eventually take into consideration when serializing self.options = kwargs def encode(self, obj): # decode back to str, as orjson returns bytes return orjson.dumps(obj).decode('utf-8') app = Flask(__name__) app.json_encoder = ORJSONEncoder app.json_decoder = ORJSONDecoder
Question: The JointPlot documentation does not show the title : can it be set? http://seaborn.pydata.org/generated/seaborn.jointplot.html?highlight=reg 

Answer: this worked for me p = sns.jointplot(x = 'x_', y = 'y_', data = df, kind="kde") p.fig.suptitle("Your title here") p.ax_joint.collections[0].set_alpha(0) p.fig.tight_layout() p.fig.subplots_adjust(top=0.95) # Reduce plot to make room
Question: I'm using flask and getting error at set_values. I'm reading the input from html and passing it to the code  @app.route('/home', methods=['POST']) def first(): source = request.files['first'] destination = request.files['second'] df = pd.read_csv(source) df1 = pd.read_csv(destination) val1 = int(request.form['val1']) val2 = int(request.form['val2']) val3 = int(request.form['val3']) target = request.form['str'] df2 = df[df.columns[val2]] count = 0 for j in df[df.columns[val1]]: x = df1.loc[df1[df1.columns[val3]] == j].index.values for i in x: df1.set_value(i, target, df2[count]) count = count + 1 df1.to_csv('result.csv', index=False) 

Answer: Check your pandas version. df.set_value() is deprecated since pandas version 0.21.0 Instead use df.at import pandas as pd df = pd.DataFrame({"A":[1, 5, 3, 4, 2], "B":[3, 2, 4, 3, 4], "C":[2, 2, 7, 3, 4], "D":[4, 3, 6, 12, 7]}) df.at[2,'B']=100 A B C D 0 1 3 2 4 1 5 2 2 3 2 3 100 7 6 3 4 3 3 12 4 2 4 4 7
Question: I'm trying to make an API POST request using the Python requests library. I am passing through an Authorization header but when I try debugging, I can see that the header is being dropped. I have no idea what's going on. Here's my code: access_token = get_access_token() bearer_token = base64.b64encode(bytes("'Bearer {}'".format(access_token)), 'utf-8') headers = {'Content-Type': 'application/json', 'Authorization': bearer_token} data = '{"FirstName" : "Jane", "LastName" : "Smith"}' response = requests.post('https://myserver.com/endpoint', headers=headers, data=data)  As you can see above, I manually set the Authorization header in the request arguments, but it is missing the actual request's headers: {'Connection': 'keep-alive', 'Content-Type': 'application/json', 'Accept-Encoding': 'gzip, deflate', 'Accept': '*/*', 'User-Agent': 'python-requests/2.4.3 CPython/2.7.9 Linux/4.1.19-v7+'}. An additional piece of information is that if I change the POST request to a GET request, the Authorization header passes through normally! Why would this library be dropping the header for POST requests and how do I get this to work? Using v2.4.3 of the requests lib and Python 2.7.9

Answer: TLDR The url you are requesting redirects POST requests to a different host, so the requests library drops the Authoriztion header in fear of leaking your credentials. To fix that you can override the responsible method in requests' Session class. Details In requests 2.4.3, the only place where reqeuests removes the Authorization header is when a request is redirected to a different host. This is the relevant code:  if 'Authorization' in headers: # If we get redirected to a new host, we should strip out any # authentication headers. original_parsed = urlparse(response.request.url) redirect_parsed = urlparse(url) if (original_parsed.hostname != redirect_parsed.hostname): del headers['Authorization']   In newer versions of requests, the Authorization header will be dropped in additional cases (for example if the redirect is from a secure to a non-secure protocol). So what probably happens in your case, is that your POST requests get redirected to a different host. The only way you can provide authentication for a redirected host using the requests library, is through a .netrc file. Sadly that will only allow you to use HTTP Basic Auth, which doesn't help you much. In that case, the best solution is probably to subclass requests.Session and override this behavior, like so: from requests import Session class NoRebuildAuthSession(Session): def rebuild_auth(self, prepared_request, response): """ No code here means requests will always preserve the Authorization header when redirected. Be careful not to leak your credentials to untrusted hosts! """ session = NoRebuildAuthSession() response = session.post('https://myserver.com/endpoint', headers=headers, data=data)  Edit I have opened a pull-request to the requests library on github to add a warning when this happens. It has been waiting for a second approval to be merged (three months already).
Question: In jupyter notebook I am running through this error. I am using py I just installed pytorch, previously it was working fine.  import pyodbc import pandas as pd import matplotlib.pyplot as plt import warnings warnings.filterwarnings('ignore')  When I run the above cell I got the following error: --------------------------------------------------------------------------- ValueError Traceback (most recent call last) <ipython-input-2-1051f624fd0e> in <module>() 1 import pyodbc ----> 2 import pandas as pd 3 import matplotlib.pyplot as plt 4 5 import warnings C:\Anaconda3\lib\site-packages\pandas\__init__.py in <module>() 29 30 try: ---> 31 from pandas._libs import hashtable as _hashtable, lib as _lib, tslib as _tslib 32 except ImportError as e: # pragma: no cover 33 # hack but overkill to use re C:\Anaconda3\lib\site-packages\pandas\_libs\__init__.py in <module>() 1 # flake8: noqa 2 ----> 3 from .tslibs import ( 4 NaT, 5 NaTType, C:\Anaconda3\lib\site-packages\pandas\_libs\tslibs\__init__.py in <module>() 1 # flake8: noqa 2 ----> 3 from .conversion import localize_pydatetime, normalize_date 4 from .nattype import NaT, NaTType, iNaT, is_null_datetimelike 5 from .np_datetime import OutOfBoundsDatetime __init__.pxd in init pandas._libs.tslibs.conversion() ValueError: numpy.ufunc size changed, may indicate binary incompatibility. Expected 216 from C header, got 192 from PyObject 

Answer: Which version of numpy do you have installed? I got the same error with 1.18.5, downgrading to 1.16.0 or 1.16.1 solved the issue. Also check which numpy version is required by pytorch, and any other library requiring numpy, I found pipdeptree quite useful for that.