eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I have an old linear model which I wish to improve using XGBoost. I have the predictions from the old model, which I wish to use as a base margin. Also, due to the nature of what I'm modeling, I need to use weights. My old glm is a poisson regression with formula number_of_defaults/exposure ~ param_1 + param_2 and weights set to exposure (same as denominator in response variable). When training the new XGBoost model on data, I do this: xgb_model = xgb.XGBRegressor(n_estimators=25, max_depth=100, max_leaves=100, learning_rate=0.01, n_jobs=4, eval_metric="poisson-nloglik", nrounds=50) model = xgb_model.fit(X=X_train, y=y_train, sample_weight=_WEIGHT, base_margin=_BASE_MARGIN)  , where _WEIGHT and _BASE_MARGIN are the weights and predictions (popped out of X_train). But how do I do cross validation or out of sample analysis when I need to specify weights and base margin? As far as I see I can use sklearn and GridSearchCV, but then I would need to specify weights and base margin in XGBRegressor() (instead of in fit() as above). The equivalent of base_margin in XGBRegressor() is the argument base_score, but there is no argument for weight. Also, I could potentially forget about doing cross-validation, and just use a training and test dataset, and I would then use eval_set argument in XGBRegressor(), but if I did that there is no way of specifying what is weight and what is base margin in the different sets. Any guidance in the right direction is much appreciated!

Answer: You can use cross_val_predict with fit_params argument, or GridSearchCV.fit with **fit_params. Here is a working proof of concept import xgboost as xgb from sklearn import datasets from sklearn.model_selection import cross_val_predict, GridSearchCV import numpy as np # Sample dataset diabetes = datasets.load_diabetes() X = diabetes.data[:150] y = diabetes.target[:150] xgb_model = xgb.XGBRegressor(n_estimators=5) fit_params = dict(sample_weight=np.abs(X[:, 0]), base_margin=np.abs(X[:, 1])) # Simple fit xgb_model.fit(X, y, **fit_params) # cross_val_predict y_pred = cross_val_predict(xgb_model, X, y, cv=3, fit_params=fit_params) print(y_pred.shape, y.shape) # grid search grid = GridSearchCV(xgb_model, param_grid={"n_estimators": [5, 10, 15]}) grid.fit(X, y, **fit_params)  You can see what happen in the code source: here, here and here. The last link is where fit_params get indexing following cross validation splits.
Question: Windows Version: Windows 10 Pro 21H2 19044.1706 GPU: rtx2070 import tensorflow as tf import torch print(torch.__version__) #1.10.1+cu113 print(torch.version.cuda) #11.3 print(tf.__version__) #2.9.1  and i run python .\object_detection\builders\model_builder_tf2_test.py  i can get 'Ran 24 tests in 18.279s OK (skipped=1)' result; But when I want to train my model, i use feature_extractor { type: 'faster_rcnn_inception_resnet_v2_keras' }  in my pipeline_config, and i run python .\object_detection\model_main_tf2.py --logtostderr --pipeline_config_path=LOCATION_OF_MY_PIPECONFIG --model_dir=LOCATION_OF_MY_MODEL_DIR  And then i get the following error  In my system environment variable , 'CUDA_DIR' is variable and can be accessed

Answer: I had the same problem and just fixed it. The library can't find the folder even if you set the "CUDA_DIR" because it's not using that variable or any other I tried. This post is helpful in understanding the issue. The only solution I was able to find is just copying the required files. Steps for a quick fix:  Find where your CUDA nvvm is installed (for me it is "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.6"). Find the working directory for your script (the environment or the directory you are running the script in). Copy the entire nvvm folder into the working directory and your script should work.  This is not a great solution but until someone else posts a answer you can at least run your code.
Question: Im getting this error with my pydantic schema, but oddly it is generating the object correctly, and sending it to the SQLAlchemy models, then it suddenly throws error for all elements in the model. response -> id field required (type=value_error.missing) response -> date field required (type=value_error.missing) response -> time field required (type=value_error.missing) response -> price field required (type=value_error.missing) response -> distance field required (type=value_error.missing) response -> origin_id field required (type=value_error.missing) response -> destination_id field required (type=value_error.missing) response -> driver_id field required (type=value_error.missing) response -> passenger_id field required (type=value_error.missing) response -> vehicle_id field required (type=value_error.missing) response -> status field required (type=value_error.missing)  i must say that all the fields should have values. And the error trace do not references any part of my code so i dont even know where to debug. Im a noob in SQLAlchemy/pydantic here are some parts of the code class Trip(BaseModel): id: int date: str time: str price: float distance: float origin_id: int destination_id: int driver_id: int passenger_id: int vehicle_id: int status: Status class Config: orm_mode = True  class TripDB(Base): __tablename__ = 'trip' __table_args__ = {'extend_existing': True} id = Column(Integer, primary_key=True, index=True) date = Column(DateTime, nullable=False) time = Column(String(64), nullable=False) price = Column(Float, nullable=False) distance = Column(Float, nullable=False) status = Column(String(64), nullable=False) origin_id = Column( Integer, ForeignKey('places.id'), nullable=False) destination_id = Column( Integer, ForeignKey('places.id'), nullable=False) origin = relationship("PlaceDB", foreign_keys=[origin_id]) destination = relationship("PlaceDB", foreign_keys=[destination_id]) driver_id = Column( Integer, ForeignKey('driver.id'), nullable=False) vehicle_id = Column( Integer, ForeignKey('vehicle.id'), nullable=False) passenger_id = Column( Integer, ForeignKey('passenger.id'), nullable=False)  def create_trip(trip: Trip, db: Session): origin = db.query(models.PlaceDB).filter(models.PlaceDB.id == trip.origin_id).first() destination = db.query(models.PlaceDB).filter(models.PlaceDB.id == trip.destination_id).first() db_trip = TripDB( id=(trip.id or None), date=trip.date or None, time=trip.time or None, price=trip.price or None, distance=trip.distance or None, origin_id=trip.origin_id or None, destination_id=(trip.destination_id or None), status=trip.status or None, driver_id=trip.driver_id or None, passenger_id=trip.passenger_id or None, vehicle_id=trip.vehicle_id or None, origin=origin, destination=destination) try: db.add(db_trip) db.commit() db.refresh(db_trip) return db_trip except: return "Somethig went wrong" 

Answer: It seems like a bug on the pydantic model, it happened to me as well, and i was not able to fix it, but indeed if you just skip the type check in the route it works fine
Question: I have a dictionary with keys and values like: my_dict = {'a':33, 'b': 'something', 'c': GETDATE(), 'd': 55}  Assume column names in the SQL table are also named like the keys of the dict, i.e. "a,b,c,d". The actual dictionary is 20+ key:value pairs. Code I have used pyodbc.connect to create a cursor which I could use to execute an SQL INSERT statement: for k in my_dict.keys(): cursor.execute( ''' INSERT INTO TABLEabc (%s) VALUES (%s) ''' % (k, my_dict[k]) )  This seems inefficient though because it's a new SQL operation each time.  What is the easiest way to insert the values using a loop? How could I write it so that it just makes one insert with all the values? 

Answer: If you're using pyodbc then this might work: columns = {row.column_name for row in cursor.columns(table='TABLEabc')} safe_dict = {key: val for key, val in my_dict.items() if key in columns} # generate a parameterised query for the keys in our dict query = "INSERT INTO TABLEabc ({columns}) VALUES ({value_placeholders})".format( columns=", ".join(safe_dict.keys()), value_placeholders=", ".join(["?"] * len(safe_dict)), ) cursor.execute(query, list(safe_dict.values()))  It is intended to be safe from SQL injection because:  we filter for only keys which are actual column names in the db we use pyodbc cursor execute params, so the values will be escaped properly  Where it possibly won't work:  if any of the column names need to be quoted and escaped, this won't happen automatically so it will fail  Quoting/escaping is db-specific so we would have to check the rules for our actual db and apply that to the dict keys that we format into the query. (or find some way to get pyodbc to do that for us, not sure if possible) If you trust your my_dict not to contain malicious code then you can simplify to just: query = "INSERT INTO TABLEabc ({columns}) VALUES ({value_placeholders})".format( columns=", ".join(my_dict.keys()), value_placeholders=", ".join(["?"] * len(my_dict)), ) cursor.execute(query, list(my_dict.values()))
Question: I'm trying to write an endpoint that just accepts an image and attempts to convert it into another format, by running a command on the system. Then I return the converted file. It's slow and oh-so-simple, and I don't have to store files anywhere, except temporarily. I'd like all the file-writing to happen in a temporary directory, so it gets cleaned up. The route works fine if the output file is not in the temporary directory. But if I try to put the output file in the temporary directory, the FileResponse can't find it, and requests fail. RuntimeError: File at path /tmp/tmpp5x_p4n9/out.jpg does not exist. Is there something going on related to the asynchronous nature of FastApi that FileResponse can't wait for the subprocess to create the file its making? Can I make it wait? (removing async from the route does not help). @app.post("/heic") async def heic(img: UploadFile): with TemporaryDirectory() as dir: inname = os.path.join(dir, "img.heic") f = open(inname,"wb") f.write(img.file.read()) f.flush() # setting outname in the temp dir fails! # outname = os.path.join(dir, 'out.jpg') outname = os.path.join('out.jpg') cmd = f"oiiotool {f.name} -o {outname}" process = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE) process.wait() return FileResponse(outname, headers={'Content-Disposition':'attachment; filename=response.csv'})  Thank you for any insights!

Answer: According to the documentation of TemporaryDirectory()  This class securely creates a temporary directory using the same rules as mkdtemp(). The resulting object can be used as a context manager (see Examples). On completion of the context or destruction of the temporary directory object, the newly created temporary directory and all its contents are removed from the filesystem.  It seems that the directory and contents are already being released before the FastAPI request is returned. However you can use Dependency injection in FastAPI, so why no inject a temporary directory? First define the dependency: async def get_temp_dir(): dir = TemporaryDirectory() try: yield dir.name finally: del dir  And add the dependency to your endpoint: @app.post("/heic") async def heic(imgfile: UploadFile = File(...), dir=Depends(get_temp_dir)): inname = os.path.join(dir, "img.heic") f = open(inname,"wb") f.write(imgfile.file.read()) f.flush() outname = os.path.join(dir, 'out.jpg') cmd = f"oiiotool {f.name} -o {outname}" process = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE) process.wait() return FileResponse(inname, headers={'Content-Disposition':'attachment; filename=response.csv'})  I've tested it with returning the incoming file and that works.
Question: I'm using BigQuery for the first time. client.list_rows(table, max_results = 5).to_dataframe();  Whenever I use to_dataframe() it raises this error:  ValueError: Please install the 'db-dtypes' package to use this function.  I found this similar problem (almost exactly the same), but I can't understand how to implement their proposed solution.

Answer: I was able to replicate your use case as shown below.  Easiest solution is to pip install db-dtypes as mentioned by @MattDMo. Or you can specify previous version of google-cloud-bigquery by creating a requirements.txt with below contents: google-cloud-bigquery==2.34.3  And then pip install by using command as shown below: pip install -r /path/to/requirements.txt  Output of my sample replication:
Question: I would like to have various dynamic "shortcuts" (rule names) in my Snakemake workflow without needing marker files. The method I have in mind is similar to eval in GNU Make, but it doesn't seem like Snakemake can evaluate variable-expanded code in the Snakefile syntax. Is there a way to accomplish this? Here's a simplified example Snakefile. I want to have a rule name corresponding to each output "stage", and right now I have to define them manually. Imagine if I had more "stages" and "steps" and wanted to have a rule that could make all "b", "d", or "z" files if I add those stages. It would be far more concise to dynamically define the rule name than to define every single combination, updated each time I add a new stage. stages = ['a', 'b'] steps = [1, 2] rule all: input: expand('{stage}{step}_file', stage=stages, step=steps) rule: output: touch('{stage}{step}_file') # Can these two be combined so that I don't have to add more # rules for each new "stage" above while retaining the shorthand # rule name corresponding to the stage? rule a: input: expand('a{step}_file', step=steps) rule b: input: expand('b{step}_file', step=steps) 

Answer: Since Snakefile is a Python file, the following might help to achieve what you are after: for stage in stages: rule: name: f'{stage}' input: expand(f'{stage}{{step}}_file', step=steps)
Question: I am trying to replace all None values in all sublists within a list to incremental numbers starting from 0 but excluding the numbers from the skip list. And there is one more requirement. If first element of a sublist matches to any other sublists' first element, then they all need to have the same value that replaces None in them if present. This is what I could try so far. skip = [1,2] a = [[1, None, 2], [3, 4, 5], [1, None, 7], [8, 9, 10],[11, None, 12]] b = 0 d = {} for i in range(len(a)): if a[i][1]==None: if b in skip: print("found b in skip") b = b + 1 if a[i][1] in d.keys(): a[i][1] = d[a[i][1]] else: a[i][1] = b d[a[i][0]] = b b = b + 1 print(d) print(a)  Output: found b in skip {1: 2, 11: 3} [[1, 0, 2], [3, 4, 5], [1, 2, 7], [8, 9, 10], [11, 3, 12]]  Expected output: [[1, 0, 2], [3, 4, 5], [1, 0, 7], [8, 9, 10], [11, 3, 12]] 

Answer: You're looking up the wrong element in the cache in a couple places, and not skipping properly when the skip list contains consecutive elements. Here's the minimal fix with inline comments indicating changed lines: skip = [1,2] a = [[1, None, 2], [3, 4, 5], [1, None, 7], [8, 9, 10],[11, None, 12]] b = 0 d = {} for i in range(len(a)): if a[i][1]==None: while b in skip: # Loop until skipped all elements in skip print("found b in skip") b = b + 1 if a[i][0] in d.keys(): # Check for first, not second element a[i][1] = d[a[i][0]] # Use first element, not second, for lookup else: a[i][1] = b d[a[i][0]] = b # Indent so we only set cached value on cache miss b = b + 1 # Indent so we only increment b on new first element print(d) print(a)  Try it online! And here's a more heavily modified version that is somewhat more Pythonic, using names, not indexing (when possible): skip = {1,2} # Use set instead of list; if skip is large, list membership check will be expensive, set will stay cheap a = [[1, None, 2], [3, 4, 5], [1, None, 7], [8, 9, 10],[11, None, 12]] b = 0 d = {} for sublist in a: # Loop over values instead of indexing a over and over, C-style (slow and less readable) first, middle, _ = sublist # Unpack to useful names (reducing risk of misindexing, and making more readable code) # Not unpacking in for loop itself because we need to reassign elements of sublist if middle is None: # Always use is/is not to compare to None while b in skip: b += 1 # Use += to avoid repeating variable name if first in d: # No need for .keys(); "in d" has same effect as "in d.keys()" and avoids creating unnecessary keys view sublist[1] = d[first] # Indexing needed for assignment to modify original list else: sublist[1] = d[first] = b # Can populate cache and reassign middle value all at once b += 1 print(d) print(a)  Try it online! Either version gets the expected output from the question.
Question: I would like to reduce space between legend columns (An example is shown in the attached image). So, what I want to do is, [before] (Sym.)A ------ (Sym.)B ------ (Sym.)C ------ (Sym.)D  [after] (Sym.)A -- (Sym.)B -- (Sym.)C -- (Sym.)D Is there a way to do it? (e.g., plt.legend(ncol=4, [a hidden parameter??])) Thanks! Image of the current output: 

Answer: Take a look at the description of columnspacing from documentation. You can try: plt.legend(ncol=4, columnspacing=0.8)
Question: I am wondering why meta data (e.g. __name__, __doc__) for the wrapped method/function by partial is not inherited by default. Instead, functools provides the update_wrapper function. Why it is not done by default is not mentioned anywhere (as far as I could see) e.g. here and many tutorials on functools.partial talk about how to "solve the issue" of a missing __name__. Are there examples where inheriting this information causes problems/confusion?

Answer: Think about what that would actually look like: def add(x, y): "Adds two numbers" return x + y add_5 = partial(add, 5)  Would it actually make sense for add_5 to have __name__ set to "add" and __doc__ set to "Adds two numbers"? The callable created by partial behaves completely differently from the original function. It wouldn't be appropriate for the new callable to inherit the name and docstring of a function with completely different behavior.
Question: I'm working on trying to compare the converge rate of SGD and GD algorithms for the neural networks. In PyTorch, we often use SGD optimizer as follows. train_dataloader = torch.utils.data.DataLoader(train_dataset, batch_size=64, shuffle=True) optimizer = torch.optim.SGD(model.parameters(), lr=0.001)  for epoch in range(epochs): running_loss = 0 for input_batch, labels_batch in train_dataloader: input = input_batch y_hat = model(input) y = labels_batch L = loss(y_hat, y) optimizer.zero_grad() L.backward() optimizer.step() running_loss += L.item()  My understanding about the optimizer here is that the SGD optimizer actually does the Mini-batch Gradient Descent algorithm because we feed the optimizer one batch of data at one time. So, if we set the batch_size parameter as the size of all data, the code actually does Gradient Descent for the neural network. Is my understanding correct?

Answer: Your understanding is correct. SGD is just updating weights based on the gradient computed by backpropagation. The flavor of gradient descent that it performs is therefore determined by the data loader.  Gradient descent (aka batch gradient descent): Batch size equal to the size of the entire training dataset. Stochastic gradient descent: Batch size equal to one and shuffle=True. Mini-batch gradient descent: Any other batch size and shuffle=True. By far the most common in practical applications.
Question: I am writing a python regex that matches only string that consists of letters, digits and one or more question marks. For example, regex1: ^[A-Za-z0-9?]+$ returns strings with or without ? I want a regex2 that matches expressions such as ABC123?A, 1AB?CA?, ?2ABCD, ???, 123? but not ABC123, ABC.?1D1, ABC(a)?1d on mysql, I did that and it works: select * from ( select * from norm_prod.skill_patterns where pattern REGEXP '^[A-Za-z0-9?]+$') AS XXX where XXX.pattern not REGEXP '^[A-Za-z0-9]+$' 

Answer: How about something like this: ^(?=.*\?)[a-zA-Z0-9\?]+$  As you can see here at regex101.com Explanation The (?=.*\?) is a positive lookahead that tells the regex that the start of the match should be followed by 0 or more characters and then a ? - i.e., there should be a ? somewhere in the match. The [a-zA-Z0-9\?]+ matches one-or-more occurrences of the characters given in the character class i.e. a-z, A-Z and digits from 0-9, and the question mark ?. Altogether, the regex first checks if there is a question mark somewhere in the string to be matched. If yes, then it matches the characters mentioned above. If either the ? is not present, or there is some foreign character, then the string is not matched.
Question: Since I'm looping over permutations I don't want too frequent updates of the progress bar, so I set miniters to be one-tenth of the total length: total_len = len(list(itertools.permutations(range(N), 2))) for row_a, row_b in tqdm(itertools.permutations(range(N), 2), total=total_len , miniters=int(total_len/10), disable=disable):    Any idea what's causing this unstable performance of miniters?

Answer: I think I found the reason, another argument maxinterval controls the maximum update interval which is set to 10 (seconds) by default, so when the progress bar updates too slowly it'll automatically modify the miniters parameter. Therefore I'll need to specify a larger maxinterval as well. So the code should be total_len = len(list(itertools.permutations(range(N), 2))) # or a larger value than 200 (seconds) for row_a, row_b in tqdm(itertools.permutations(range(N), 2), total=total_len , miniters=int(total_len/10), maxinterval=200, disable=disable):
Question: I have the following example code that works with a regular map def f(x_y): x, y = x_y return x.sum() + y.sum() xs = [jnp.zeros(3) for i in range(4)] ys = [jnp.zeros(2) for i in range(4)] list(map(f, zip(xs, ys))) # returns: [DeviceArray(0., dtype=float32), DeviceArray(0., dtype=float32), DeviceArray(0., dtype=float32), DeviceArray(0., dtype=float32)]  How can I use jax.vmap instead? The naive thing is: jax.vmap(f)(zip(xs, ys))  but this gives: ValueError: vmap was requested to map its argument along axis 0, which implies that its rank should be at least 1, but is only 0 (its shape is ()) 

Answer: For using jax.vmap, you do not need to zip your variables. You can write what you want like below: import jax.numpy as jnp from jax import vmap def f(x_y): x, y = x_y return x.sum() + y.sum() xs = jnp.zeros((4,3)) ys = jnp.zeros((4,2)) vmap(f)((xs, ys))  Output: DeviceArray([0., 0., 0., 0.], dtype=float32)