eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I'm new to this rdkit, below is the code that I'm using to get the chemical image from the formula, from rdkit import Chem m = Chem.MolFromSmiles('OCC1OC(C(C(C1O)O)O)[C]1(C)(CO)CC(=O)C=C(C1CCC(=O)C)C') m  if the code is correct, it displays the structre. The above code displays the error saying "[15:23:55] Explicit valence for atom # 11 C, 5, is greater than permitted"  I tried adding try catch but, it does not help. I need to catch this exception/Message and display it to the user. How can we do this?Issue Image Here try: from rdkit import Chem m = Chem.MolFromSmiles('OCC1OC(C(C(C1O)O)O)[C]1(C)(CO)CC(=O)C=C(C1CCC(=O)C)C') m except Exception as e: print(e) 

Answer: It is not an Error, it is a Warning. Your code does not break doing that assignment. So there is no Error to catch. But your solution lays within the return of Chem.MolFromSmiles. If it fails to build a mol object of your SMILES it returns None, whereas it returns the mol object when it manages to "deal with the SMILES properly". For sure you can raise an error yourself! from rdkit import Chem m = Chem.MolFromSmiles('OCC1OC(C(C(C1O)O)O)[C]1(C)(CO)CC(=O)C=C(C1CCC(=O)C)C') if m is None: print("hey user, there was an error!") # or even raise an error: if m is None: raise ValueError("Could not interpret SMILES to mol object")  I know my answer is lacking. My solution is not forwarding the warning to the user. I feel like it is quite hard to do. If there is a solution it might be found utilizing logging or warnings package. I would also be interested in that but couldn't manage so far myself.
Question: I am trying to write a middleware in my FastAPI application, so that requests coming to endpoints matching a particular format will be rerouted to a different URL, but I am unable to find a way to do that since request.url is read-only. I am also looking for a way to update request headers before rerouting. Are these things even possible in FastAPI? Redirection is the best I could do so far: from fastapi import Request from fastapi.responses import RedirectResponse @app.middleware("http") async def redirect_middleware(request: Request, call_next): if matches_certain_format(request.url.path): new_url = create_target_url(request.url.path) return RedirectResponse(url=new_url) 

Answer: To change the request's URL path—in other words, reroute the request to a different endpoint—one can simply modify the request.scope['path'] value inside the middleware, before processing the request, as demonstrated in Option 3 of this answer. If your API endpoints include path parameters (e.g., '/users/{user_id}'), then you mgiht want to have a look at this answer on how to extract that kind of path from the request object, and then compare it against a predefined list of routes_to_reroute, as shown below. As for updating the request headers, or adding new custom headers to the request, you can follow a similar approach to the one described here, which demonstrates how to modify the request.scope['headers'] value. Working Example If you would like to avoid maintaining a list of routes to reroute and performing checks inside the middleware, you could instead mount a sub-application, which will contain only the routes that require rerouting, and add the middleware to that sub-app, similar to Option 3 of this answer. from fastapi import FastAPI, Request app = FastAPI() routes_to_reroute = ['/'] @app.middleware('http') async def some_middleware(request: Request, call_next): if request.url.path in routes_to_reroute: request.scope['path'] = '/welcome' headers = dict(request.scope['headers']) headers[b'custom-header'] = b'my custom header' request.scope['headers'] = [(k, v) for k, v in headers.items()] return await call_next(request) @app.get('/') async def main(): return 'OK' @app.get('/welcome') async def welcome(request: Request): return {'msg': 'Welcome!', 'headers': request.headers}
Question: I tried running the following: from google.cloud import bigquery  But from some reason, I keep getting this "Bases must be types" error: --------------------------------------------------------------------------- TypeError Traceback (most recent call last) <ipython-input-3-1035661e8528> in <module> ----> 1 from google.cloud import bigquery /opt/conda/lib/python3.7/site-packages/google/cloud/bigquery/__init__.py in <module> 33 __version__ = bigquery_version.__version__ 34 ---> 35 from google.cloud.bigquery.client import Client 36 from google.cloud.bigquery.dataset import AccessEntry 37 from google.cloud.bigquery.dataset import Dataset /opt/conda/lib/python3.7/site-packages/google/cloud/bigquery/client.py in <module> 58 59 import google.api_core.client_options ---> 60 import google.api_core.exceptions as core_exceptions 61 from google.api_core.iam import Policy 62 from google.api_core import page_iterator /opt/conda/lib/python3.7/site-packages/google/api_core/exceptions.py in <module> 27 import warnings 28 ---> 29 from google.rpc import error_details_pb2 30 31 try: /opt/conda/lib/python3.7/site-packages/google/rpc/error_details_pb2.py in <module> 18 # source: google/rpc/error_details.proto 19 """Generated protocol buffer code.""" ---> 20 from google.protobuf import descriptor as _descriptor 21 from google.protobuf import descriptor_pool as _descriptor_pool 22 from google.protobuf import message as _message /opt/conda/lib/python3.7/site-packages/google/protobuf/descriptor.py in <module> 38 import warnings 39 ---> 40 from google.protobuf.internal import api_implementation 41 42 _USE_C_DESCRIPTORS = False /opt/conda/lib/python3.7/site-packages/google/protobuf/internal/api_implementation.py in <module> 102 try: 103 # pylint: disable=g-import-not-at-top --> 104 from google.protobuf.pyext import _message 105 sys.modules['google3.net.proto2.python.internal.cpp._message'] = _message 106 _c_module = _message TypeError: bases must be types  I'm not sure what I did wrong. Maybe I install or uninstalled a package somewhere. But now I can't get this simple command to work. Any help would be greatly appreciated.

Answer: Marking this as an answer for better visibility, as this resolved the question: pip installing protobuff v3.20.1 as @runner16 mentioned  pip install protobuf==3.20.1
Question: I have a very long generator function that I want to process as a column using Polars. Due to its size, I want to run it in lazy streaming mode using the generator as a source, but I have been unable to work out how to do it (if it is possible). Creating a normal dataframe and then converting to lazy obviously doesn't work since the generator is exhausted before the lazy plan is run with collect(). This also happens with the LazyFrame initialiser, which is just a shortcut to above. Are there any other options that don't involve writing then scanning a csv? Example code: import polars as pl def Generator(): yield 1 yield 2 yield 3 generator = Generator() df = pl.DataFrame({"a": generator}).lazy() print(df) # naive plan... print([i for i in generator]) # [] generator2 = Generator() df = pl.LazyFrame({"a": generator2}) print(df) # naive plan... print([i for i in generator2]) # [] 

Answer:  Creating a normal dataframe and then converting to lazy obviously doesn't work since the generator is exhausted before the lazy plan is run with collect().  This is a mistaken assumption. If you do: df = pl.DataFrame({"a": generator}).lazy()  then, yes, the generator is exhausted but all of its data is in df so it doesn't matter that the generator is exhausted If you take that back a step and imagine you're just doing df = pl.DataFrame({"a": generator}) then clearly you've got the data in df. When you tack on .lazy() all you're doing is setting a flag that any operations shouldn't be computed until .collect (or similar) is invoked. If you've got the memory to do that then it is the best way to consume the generator. If you don't have the memory to consume the generator then you'd have to dump it into a file but that file doesn't have to be a csv. You can use pyarrow to dump it into a parquet or ipc file which you can subsequently scan with polars. That would look like this: import pyarrow as pa import polars as pl generator = Generator() with pa.ipc.new_file('somefile.ipc', pa.schema([('a',pa.int32())])) as writer: while True: try: writer.write( pa.record_batch([pa.array([next(generator)], pa.int32())], names=['a']) ) except StopIteration: break df=pl.scan_ipc('somefile.ipc')  If you want to write a parquet file then you'd use import pyarrow.parquet as pq import pyarrow as pa import polars as pl generator = Generator() with pq.ParquetWriter('somefile.parquet', pa.schema([('a',pa.int32())])) as writer: while True: try: writer.write( pa.record_batch([pa.array([next(generator)], pa.int32())], names=['a']) ) except StopIteration: break dfpq=pl.scan_parquet('somefile.parquet')
Question: Loss functions in pytorch use "mean" reduction. So it means that the model gradient will have roughly the same magnitude given any batch size. It makes sense that you want to scale the learning rate up when you increase batch size because your gradient doesn't become bigger as you increase batch size. For gradient accumulation in PyTorch, it will "sum" the gradient N times where N is the number of times you call backward() before you call step(). My intuition is that this would increase the magnitude of the gradient and you should reduce the learning rate, or at least not increase it. But I saw people wrote multiplication to gradient accumulation steps in this repo: if args.scale_lr: args.learning_rate = ( args.learning_rate * args.gradient_accumulation_steps * args.train_batch_size * accelerator.num_processes )  I also see similar code in this repo: model.learning_rate = accumulate_grad_batches * ngpu * bs * base_lr  I understand why you want to increase the learning rate by batch size. But I don't understand why they try to increase the learning rate by the number of accumulation steps.  Do they divide the loss by N to reduce the magnitude of the gradient? Otherwise why do they multiply learning rate by the accumulation steps? How are gradients from different GPUs accumulated? Is it using mean or sum? If it's sum, why are they multiplying the learning rate by nGPUs? 

Answer: I found that they indeed divided the loss by N (number of gradient accumulation steps). You can see sample code from accelerate package here: https://huggingface.co/docs/accelerate/usage_guides/gradient_accumulation Notice the following line of code from the guide above: loss = loss / gradient_accumulation_steps This is why you need to multiply the learning rate by gradient accumulation steps to cancel the above division. I assume that the same procedure also happens in PyTorch Lightning. I asked a related Lightning question at the github discussion here: https://github.com/Lightning-AI/lightning/discussions/17035 I hope that someone will answer later that Trainer in Lightning does the same division process. The evidence from accelerate package made me think that gradients from different GPUs are also averaged, not summed. If they are going to be summed, the loss on each GPU has to be divided by the number of GPUs. This leads to a simple intuition about gradients in most PyTorch training workflows: No matter how big or small the batch is, the gradient will always have roughly the same magnitude. If you check the magnitude of the gradient right before step() call, it should stay roughly the same even if you vary batch size, number of gradient accumulation steps, number of GPUs, or even number of computers.
Question: I am working in DuckDB in a database that I read from json. Here is the json: [{ "account": "abcde", "data": [ { "name": "hey", "amount":1, "flow":"INFLOW" }, { "name": "hello", "amount":-2, "flow": null } ] }, { "account": "hijkl", "data": [ { "name": "bonjour", "amount":1, "flow":"INFLOW" }, { "name": "hallo", "amount":-3, "flow":"OUTFLOW" } ] } ]  I am opening it in Python as follows: import duckdb duckdb.sql(""" CREATE OR REPLACE TABLE mytable AS SELECT * FROM "example2.json" """)  This all works fine and I get a copy of my table, but then I try to update it: duckdb.sql(""" UPDATE mytable SET data = NULL WHERE account = "abcde" """)  which crashes with --------------------------------------------------------------------------- BinderException Traceback (most recent call last) Cell In[109], line 1 ----> 1 duckdb.sql(""" 2 UPDATE mytable SET data = NULL WHERE account = "abcde" 3 """) 6 # duckdb.sql(""" 7 # DELETE FROM mytable WHERE account = "abcde" 8 # """) 10 duckdb.sql(""" 11 SELECT * FROM mytable 12 """) BinderException: Binder Error: Referenced column "abcde" not found in FROM clause! Candidate bindings: "mytable.data" LINE 2: ...mytable SET data = NULL WHERE account = "abcde" ^  I have searched the documentation and the error but I just can't find what I am doing wrong here.

Answer: I actually solved the issue. I had to use single quotes ' instead of double quotes " in the string comparison... Solution duckdb.sql(""" UPDATE mytable SET data = NULL WHERE account = 'abcde' """)  correctly does ┌─────────┬──────────────────────────────────────────────────────────────────────────────────────────────────┐ │ account │ data │ │ varchar │ struct("name" varchar, amount bigint, flow varchar)[] │ ├─────────┼──────────────────────────────────────────────────────────────────────────────────────────────────┤ │ hijkl │ [{'name': bonjour, 'amount': 1, 'flow': INFLOW}, {'name': hallo, 'amount': -3, 'flow': OUTFLOW}] │ │ abcde │ NULL │ └─────────┴──────────────────────────────────────────────────────────────────────────────────────────────────┘  Interestingly, ChatGPT helped me spot this mistake. (There is a ban on posting AI answers, but it's OK if they are human-verified).
Question: I am using numba and I would like to know if an overflow has occurred when I multiply two integers. Say the integers are positive for simplicity. I have written the following function to try and achieve this:  from numba import njit import numpy as np @njit def safe_mul(a, b): c = a * b print('a: ', a) print('b: ', b) print('c: ', c) print('c//a: ', c//a) print('0//a: ', 0//a) if c // a != b or c // b != a: # do something else or raise error raise ValueError() return c @njit def safe_mul_2(a, b): if (np.log2(np.abs(a)) + np.log2(np.abs(b))) >= 63: \# do something else or raise error raise ValueError() return a * b print(safe_mul(2**21, 2**51))  The code prints:  a: 2097152 b: 2251799813685248 c: 0 c//a: 2251799813685248 0//a: 0 0  safe_mul does not catch the overflow when 2**21 and 2**51 are passed in. Perhaps numba is compiling out the integer division since it knows c has just been multiplied by what it is being divided by? I am not sure about this since when you enter something where the arguments are not both powers of 2 then the error is caught. safe_mul_2 does catch the error, and is surprisingly not much slower than safe_mul (when the prints are removed). I would like to know what is happening in safe_mul and if something that is faster than safe_mul_2 can be written.

Answer: On the Numba discourse site, the user sschaer was able to provide a very fast solution. See this: https://numba.discourse.group/t/catch-overflow-in-integer-multiplication/1827 for the original discussion. The solution is copied here with sschaer's permission. LLVM has integer operations that return an overflow bit and they are already known to Numba's intermediate representation. The following functions exposes the multiplication operations: import numpy as np from numba import njit, types from numba import TypingError from numba.extending import intrinsic @intrinsic def mul_with_overflow(typingctx, a, b): if not (isinstance(a, types.Integer) and isinstance(b, types.Integer)): raise TypingError("both arguments must be integers") if a.signed != b.signed: raise TypingError("can only multiply integers of equal signedness") if a.signed: ext = lambda builder, a, b: builder.sext(a, b) mul = lambda builder, a, b: builder.smul_with_overflow(a, b) else: ext = lambda builder, a, b: builder.zext(a, b) mul = lambda builder, a, b: builder.umul_with_overflow(a, b) retint_ty = max(a, b, key=lambda ty: ty.bitwidth) sig = types.Tuple([retint_ty, types.boolean])(a, b) def codegen(context, builder, signature, args): int_ty = context.get_value_type(retint_ty) a = ext(builder, args[0], int_ty) b = ext(builder, args[1], int_ty) prod_and_flag = mul(builder, a, b) return prod_and_flag return sig, codegen @njit def foo(a, b): return mul_with_overflow(a, b) foo(np.int8(1), np.int32(-2)) # returns int32 foo(np.uint8(1), np.uint16(2)) # returns uint16 foo(np.int32(2), np.int64(np.iinfo(np.int64).max)) # overflow foo(np.uint64(1), np.int64(2)) # error foo(np.uint64(1), np.float32(2)) # error
Question: In a practical sense, what does the grid_size parameter do for you? When/why would you change it from the default? I understand from testing that it imposes a discretization on the coordinates of the resulting geometries, e.g. with grid_size=0.01 the fractional part of the coordinates will be multiples of 0.01. Does this play into the logic of the algorithms or is just a convenience for applications where, in the end, the user is going to discretize the coordinates anyway?

Answer: A practical reason why I use it sometimes is to avoid having slivers after applying overlays (in non-topological data). The code sample below illustrates this:  the intersection between the 2 polygons without grid_size results in a narrow polygon as intersection. the intersection between the 2 polygons with grid_size results in a line, which is easy to filter out.  import shapely import shapely.plotting import matplotlib.pyplot as plt poly1 = shapely.Polygon([(0, 0), (0, 10), (10, 10), (5, 0), (0, 0)]) poly2 = shapely.Polygon([(5, 0), (8, 7), (10, 7), (10, 0), (5, 0)]) intersection_nogridsize = poly1.intersection(poly2) intersection_gridsize = poly1.intersection(poly2, grid_size=1) shapely.plotting.plot_polygon(poly1, color="green") shapely.plotting.plot_polygon(poly2, color="blue") shapely.plotting.plot_polygon(intersection_nogridsize, color="red") plt.show() shapely.plotting.plot_polygon(poly1, color="green") shapely.plotting.plot_polygon(poly2, color="blue") shapely.plotting.plot_line(intersection_gridsize, color="red") plt.show()  Result without gridsize (red intersection is a polygon):  Result with gridsize (red intersection is a line):
Question: According to the docs (in Python 3.8):  By default, object implements __eq__() by using is, returning NotImplemented in the case of a false comparison: True if x is y else NotImplemented.  And also:  The correspondence between operator symbols and method names is as follows: [...] x==y calls x.__eq__(y)  So I expect  == to be equivalent to __eq__() and a custom class without an explicitly defined __eq__ to return NotImplemented when using == to compare two different instances of the class. Yet in the following, == comparison returns False, while __eq__() returns NotImplemented:  class Dummy(): def __init__(self, a): self.a = a d1 = Dummy(3) d2 = Dummy(3) d1 == d2 # False d1.__eq__(d2) # NotImplemented  Why?

Answer: == calls the right operand's .__eq__() if the left operand's .__eq__() returns NotImplemented and if both sides return NotImplemented, == will return false. You can see this behavior by changing your class: class Dummy(): def __eq__(self, o): print(f'{id(self)} eq') return NotImplemented d1 = Dummy() d2 = Dummy() print(d1 == d2) # 2180936917136 eq # 2180936917200 eq # False  This is a common behavior for operators where Python test if the left operand have an implementation, if not, Python call the same or reflection (e.g. if x.__lt__(y) is not implemented y.__gt__(x) is called) operator of the right operand. class Dummy: def __eq__(self, o): print(f"{id(self)} eq") return NotImplemented def __lt__(self, o): """reflexion of gt""" print(f"{id(self)} lt") return NotImplemented def __gt__(self, o): """reflexion of lt""" print(f"{id(self)} gt") return False def __le__(self, o): """reflexion of ge""" print(f"{id(self)} le") return NotImplemented def __ge__(self, o): """reflexion of le""" print(f"{id(self)} ge") return False d1 = Dummy() d2 = Dummy() print(d1 == d2) # 2480053379984 eq # 2480053380688 eq # False print(d1 < d2) # 2480053379984 lt # 2480053380688 gt # False print(d1 <= d2) # 2480053379984 le # 2480053380688 ge # False  ! Exception to the left before right:  If the operands are of different types, and right operand’s type is a direct or indirect subclass of the left operand’s type, the reflected method of the right operand has priority, otherwise the left operand’s method has priority.
Question: I'm using dash to build a dashboard where I am creating a unique url whenever a particular data point is clicked, how can I redirect users to this created url? I'm using the below given code where whenever someone click on any data points, click event will trigger and callback function executes. app.layout = html.Div(children=[ html.H1(children='Plot'), dcc.Graph(id='example-graph', figure = plot()), html.Div(id='dummy-div') ]) @app.callback(Output('dummy-div', 'childern'), Input('graph', 'clickData')) def redirect_to_url(clickData): triggered_id = dash.callback_context.triggered[0]['prop_id'].split('.')[0] if triggered_id=='graph' and clickData is not None: url = 'https:www.example.com' # need to develop to a method to redirect to this url 

Answer: With a clienside callback, using window.location : app.clientside_callback( """ function(clickData) { if (clickData?.points?.length) { const point = clickData['points'][0]; const path = point.customdata ?? 'fallbackPath'; const url = `https://example.com/${path}`; window.location = url; } } """, Output('dummy-div', 'children'), Input('example-graph', 'clickData'), prevent_initial_call=True # )  Nb. This option also allows to open the target page in a new tab, for this just replace the line window.location = url with : window.open(url, '_blank');   Or, by adding a dcc.Location() component in the layout, you can then update its href property using a basic callback : app.layout = html.Div(children=[ dcc.Location(id='location'), html.H1(children='Plot'), dcc.Graph(id='example-graph', figure=plot()), html.Div(id='dummy-div') ]) @app.callback(Output('location', 'href'), Input('example-graph', 'clickData'), prevent_initial_call=True) def redirect_to_url(clickData): # do something with clickData url = 'https:www.example.com' return url  A few things to note :  You don't need to check the trigger id if you have only one Input() (ie. the condition if triggered_id=='graph' is not necessary) Ensure your Input/Output ids match what you have in the layout ('graph' vs 'example-graph') Redirecting the user "whenever a particular data point is clicked" implies the condition clickData is not None instead of clickData is None
Question: I like how duckdb lets me query DataFrames as if they were sql tables: df = pandas.read_parquet("my_data.parquet") con.query("select * from df limit 10").fetch_df()  I also like how duckdb has metadata commands like SHOW TABLES;, like a real database. However, SHOW TABLES; doesn't show pandas DataFrames or other table-like objects. my question is: does duckdb offer something like SHOW TABLES; that includes both (1) real database tables and (2) table-like objects (e.g. pandas DataFrames) and their schemas? Thanks!

Answer: You can use the different metadata table functions duckdb_% as referred here For an equivalent of SHOW TABLES and convert it as a pandas dataframe import duckdb df = duckdb.sql("SELECT * FROM duckdb_tables;").df() print(df.dtypes) database_name object database_oid int64 schema_name object schema_oid int64 table_name object table_oid int64 internal bool temporary bool has_primary_key bool estimated_size int64 column_count int64 index_count int64 check_constraint_count int64 sql object dtype: object  Note : I'm using the latest version of duckDB v0.7.1