internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: searxng/searxng

Issue:
Yahoo: Result titles getting mixed together
**Version of SearXNG, commit number if you are using on master branch and stipulate if you forked SearXNG**
2024.4.19+0f70e4d30

**How did you install SearXNG?**
searxng-docker

**What happened?**
I've noticed that result titles are getting mixed up together when they shouldn't.

**How To Reproduce**
Search `!yh gba custom bios`.

**Expected behavior**
Result titles not being mixed together.

**Screenshots & Logs**
Highlighted in red are the parts where the title is wrong.
![Screenshott](https://github.com/searxng/searxng/assets/129474298/4e3dc27b-3e97-438b-a349-3e6ce317cab9)



Discussion:

Resolving patch:
diff --git a/searx/engines/yahoo.py b/searx/engines/yahoo.py
--- a/searx/engines/yahoo.py
+++ b/searx/engines/yahoo.py
@@ -132,7 +132,8 @@ def response(resp):
             continue
         url = parse_url(url)
 
-        title = extract_text(result.xpath('.//h3//a/@aria-label'))
+        title = eval_xpath_getindex(result, './/h3//a/@aria-label', 0, default='')
+        title = extract_text(title)
         content = eval_xpath_getindex(result, './/div[contains(@class, "compText")]', 0, default='')
         content = extract_text(content, allow_none=True)
 


Test patch:
Repository: searxng/searxng

Issue:
Google: Improve support for results with missing keywords
**Is your feature request related to a problem? Please describe.**
<!-- A clear and concise description of what the problem is. Ex. I'm always frustrated when [...] -->
Yes, result descriptions that come from Google include the "Missing: [Keyword] | Show results with: [Keyword]" but there is currently no way to click on the option to show results with said keyword.

**Describe the solution you'd like**

- Show "Missing: [Keyword] | Show results with: [Keyword]" on a new line and make it more obvious it's a missing keyword and not part of the description
- Have the ability to click on the "Show results with: **[Keyword]**" to do what it says it does.

**Describe alternatives you've considered**

1. Remove the part with "Show results with: **[Keyword]**" and keep the "Missing: **[Keyword]**", putting it on a new line.
2. Remove the result entirely (not ideal, since the result may be still relevant)

**Additional context**
![image](https://github.com/searxng/searxng/assets/129474298/be2df41e-42ca-4b39-b557-2a5d57b6543e)



Discussion:

Resolving patch:
diff --git a/searx/engines/google.py b/searx/engines/google.py
--- a/searx/engines/google.py
+++ b/searx/engines/google.py
@@ -62,7 +62,7 @@
 results_xpath = './/div[contains(@jscontroller, "SC7lYd")]'
 title_xpath = './/a/h3[1]'
 href_xpath = './/a[h3]/@href'
-content_xpath = './/div[@data-sncf]'
+content_xpath = './/div[@data-sncf="1"]'
 
 # Suggestions are links placed in a *card-section*, we extract only the text
 # from the links not the links itself.


Test patch:
Repository: google/turbinia

Issue:
Save_evidence should be called only when copyable=True
https://github.com/google/turbinia/blob/master/turbinia/workers/__init__.py#L123


Discussion:

Resolving patch:
diff --git a/turbinia/workers/__init__.py b/turbinia/workers/__init__.py
--- a/turbinia/workers/__init__.py
+++ b/turbinia/workers/__init__.py
@@ -120,7 +120,8 @@ def close(self, task, success, status=None):
     for evidence in self.evidence:
       if evidence.local_path:
         self.saved_paths.append(evidence.local_path)
-        task.output_manager.save_evidence(evidence, self)
+        if evidence.copyable and not config.SHARED_FILESYSTEM:
+          task.output_manager.save_evidence(evidence, self)
       if not evidence.request_id:
         evidence.request_id = self.request_id
 


Test patch:
Repository: google/turbinia

Issue:
Set version in only one place
Right now the version is set in setup.py and in turbinia/__init__.py.  We should have it set in one place so we don't need to update it in both.


Discussion:
I'll take a shot at this. Any preference as to how this is consolidated? I've seen: 

* the version stored in the `__init__.py` and read from `setup.py` using relative pathing
* a plaintext file containing the version information, accessed by both `__init__.py` and `setup.py`. 

I can do something else as well, just looking for what's preferred. 

I'll review contribution details in the meantime.

Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -18,6 +18,8 @@
 
 from __future__ import unicode_literals
 
+import sys
+
 from setuptools import find_packages
 from setuptools import setup
 
@@ -29,7 +31,10 @@
   from pip.req import parse_requirements
 
 
-turbinia_version = '20181004'
+# make sure turbinia is in path
+sys.path.insert(0, '.')
+
+import turbinia  # pylint: disable=wrong-import-position
 
 turbinia_description = (
   'Turbinia is an open-source framework for deploying, managing, and running'
@@ -41,7 +46,7 @@
 
 setup(
   name='turbinia',
-  version=turbinia_version,
+  version=turbinia.__version__,
   description='Automation and Scaling of Digital Forensics Tools',
   long_description=turbinia_description,
   license='Apache License, Version 2.0',
diff --git a/turbinia/__init__.py b/turbinia/__init__.py
--- a/turbinia/__init__.py
+++ b/turbinia/__init__.py
@@ -14,7 +14,7 @@
 # limitations under the License.
 """Main Turbinia application."""
 
-VERSION = '20181004'
+__version__ = '20181004'
 
 
 class TurbiniaException(Exception):
diff --git a/turbinia/turbiniactl.py b/turbinia/turbiniactl.py
--- a/turbinia/turbiniactl.py
+++ b/turbinia/turbiniactl.py
@@ -33,7 +33,7 @@
 from turbinia import config
 from turbinia.config import logger
 from turbinia import evidence
-from turbinia import VERSION
+from turbinia import __version__
 from turbinia.message import TurbiniaRequest
 
 log = logging.getLogger('turbinia')
@@ -90,7 +90,7 @@ def main():
       '-V',
       '--version',
       action='version',
-      version=VERSION,
+      version=__version__,
       help='Show the version')
   parser.add_argument(
       '-D',


Test patch:
Repository: google/turbinia

Issue:
Programatically setting config file
I am trying to write tests for the dftimewolf turbinia module. Loading the configuration on a system that doesn't have one will fail, so I am trying to manually feed in a test config data to see if the module behaves as expected.

I tried setting the `TURBINIA_CONFIG_PATH` environment variable, but this just *adds* the path the list of possible config paths. This would work in a pristine test environment, but it will break in my dev setup where I already have a production turbinia config file set up.

What do you think of giving `TURBINIA_CONFIG_PATH` environment variable precedence over the other potential config locations?


Discussion:
Sure, yeah, I think that should be fine.  LMK if you want to create a PR for this, otherwise I might get to it at the end of the week.
I'll send a PR

Resolving patch:
diff --git a/turbinia/config/__init__.py b/turbinia/config/__init__.py
--- a/turbinia/config/__init__.py
+++ b/turbinia/config/__init__.py
@@ -85,12 +85,16 @@ def LoadConfig():
   if CONFIG:
     return CONFIG
 
+  # If the environment variable is set, take precedence over the pre-defined
+  # CONFIGPATHs.
+  configpath = CONFIGPATH
   if ENVCONFIGVAR in os.environ:
-    CONFIGPATH.extend(os.environ[ENVCONFIGVAR].split(':'))
+    configpath = os.environ[ENVCONFIGVAR].split(':')
+
 
   config_file = None
   # Load first file found
-  for _dir, _file in itertools.product(CONFIGPATH, CONFIGFILES):
+  for _dir, _file in itertools.product(configpath, CONFIGFILES):
     if os.path.exists(os.path.join(_dir, _file)):
       config_file = os.path.join(_dir, _file)
       break


Test patch:
Repository: google/turbinia

Issue:
turbinia_job_graph.py doesn't support new job manager
turbinia_job_graph.py needs to be updated to support the new job manager (from #257).


Discussion:

Resolving patch:
diff --git a/tools/turbinia_job_graph.py b/tools/turbinia_job_graph.py
--- a/tools/turbinia_job_graph.py
+++ b/tools/turbinia_job_graph.py
@@ -19,7 +19,7 @@
 import argparse
 import graphviz
 
-from turbinia.jobs import get_jobs as turbinia_jobs
+from turbinia.jobs import manager as jobs_manager
 
 
 def create_graph():
@@ -29,15 +29,15 @@ def create_graph():
     Instance of graphviz.dot.Digraph
   """
   dot = graphviz.Digraph(comment='Turbinia Evidence graph', format='png')
-  for job in turbinia_jobs():
-    dot.node(job.name)
+  for _, job in jobs_manager.JobsManager.GetJobs():
+    dot.node(job.NAME)
     for evidence in job.evidence_input:
       dot.node(evidence.__name__, shape='box')
-      dot.edge(evidence.__name__, job.name)
+      dot.edge(evidence.__name__, job.NAME)
 
     for evidence in job.evidence_output:
       dot.node(evidence.__name__, shape='box')
-      dot.edge(job.name, evidence.__name__)
+      dot.edge(job.NAME, evidence.__name__)
   return dot
 
 


Test patch:
Repository: google/turbinia

Issue:
turbiniactl -a status breaks on local installations
When running `turbiniactl -a status` with local tasks, there is an error, and a stack trace gets thrown here and `saved_paths` is incorrectly set to None:
https://github.com/google/turbinia/blob/b65bc0d26f635655ad03ece4d65fae2e2e224915/turbinia/client.py#L250

FYI @ericzinnikas 





Discussion:
I thought this might have been a byproduct of using Redis and its default values, but I just saw this error as well when using Datastore.  Should be a quick fix.

Resolving patch:
diff --git a/turbinia/client.py b/turbinia/client.py
--- a/turbinia/client.py
+++ b/turbinia/client.py
@@ -282,6 +282,8 @@ def format_task_status(
                 task.get('name'), task.get('user'), task.get('worker_name'),
                 success, status))
         saved_paths = task.get('saved_paths', [])
+        if saved_paths is None:
+          saved_paths = []
         for path in saved_paths:
           results.append('\t{0:s}'.format(path))
       else:


Test patch:
Repository: google/turbinia

Issue:
Error when worker version doesn't match server
We'll have to add version info to TurbiniaTask so that workers can reject tasks when there is a mis-match between the versions used by the server and the workers.  Strange errors can occur when the workers are running a different version from the server.


Discussion:
@aarontp Hi! I'll want to take this task. Let me ask a couple of things first:

1) We need a version in `workers.TurbiniaTask` should this version be `turbinia.__version__`?
2) This version could be just an attribute inside `workers.TurbiniaTask` class?
3) This task requires to create the code inside the workers in order to reject the requests that have version mis-match?

Just making sure I'm in the right track before I start making code.
@fpiedrah Great, thanks much!

1. Yes, `turbinia.__version__` should be the common version referenced by all components.
2. Yes, that sounds good.
3. Also Yes: I think it should probably happen somewhere near the start of the `TurbiniaTask.run_wrapper()`.  Note that for the error to propagate back to the server, it will need to be returned as part of the `TurbiniaTaskResult()`, so it should happen after the result is created, but before the `TurbiniaTask.run()` method is called.  It would be good if the self.result.status also indicated the error (so it shows up in the client output).

I think clarifying and communicating on this before you started was a good idea, so thanks for doing that, :).

Resolving patch:
diff --git a/turbinia/workers/__init__.py b/turbinia/workers/__init__.py
--- a/turbinia/workers/__init__.py
+++ b/turbinia/workers/__init__.py
@@ -26,6 +26,7 @@
 import subprocess
 import traceback
 import uuid
+import turbinia
 
 import filelock
 
@@ -245,6 +246,7 @@ def __init__(
     self.state_key = None
     self.stub = None
     self.tmp_dir = None
+    self.turbinia_version = turbinia.__version__
     self.user = user if user else getpass.getuser()
     self._evidence_config = {}
 
@@ -429,6 +431,18 @@ def run_wrapper(self, evidence):
       try:
         self.result = self.setup(evidence)
         original_result_id = self.result.id
+
+        if self.turbinia_version != turbinia.__version__:
+          msg = 'Worker V-{0:s} and server V-{1:s} version do not match'.format(
+            self.turbinia_version,
+            turbinia.__version__
+          )
+          log.error(msg)
+          self.result.log(msg)
+          self.result.set_error(msg)
+          self.result.status = msg
+          return self.result
+
         self._evidence_config = evidence.config
         self.result = self.run(evidence, self.result)
       # pylint: disable=broad-except


Test patch:
Repository: google/turbinia

Issue:
Remove -C option
Right now when running in local mode, the -C flag is required on every turbiniactl invocation.  We should be able to look this up in the config so that isn't required.


Discussion:

Resolving patch:
diff --git a/turbinia/turbiniactl.py b/turbinia/turbiniactl.py
--- a/turbinia/turbiniactl.py
+++ b/turbinia/turbiniactl.py
@@ -85,10 +85,6 @@ def main():
   parser.add_argument(
       '-S', '--server', action='store_true',
       help='Run Turbinia Server indefinitely')
-  parser.add_argument(
-      '-C', '--use_celery', action='store_true',
-      help='Pass this flag when using Celery/Kombu for task queuing and '
-      'messaging (instead of Google PSQ/pubsub)')
   parser.add_argument(
       '-V', '--version', action='version', version=__version__,
       help='Show the version')
@@ -304,7 +300,7 @@ def main():
   # Client
   config.LoadConfig()
   if args.command not in ('psqworker', 'server'):
-    if args.use_celery:
+    if config.TASK_MANAGER.lower() == 'celery':
       client = TurbiniaCeleryClient()
     elif args.run_local:
       client = TurbiniaClient(run_local=True)
@@ -313,9 +309,8 @@ def main():
   else:
     client = None
 
-  server_flags_set = args.server or args.use_celery or args.command == 'server'
-  worker_flags_set = (
-      args.use_celery or args.command in ('psqworker', 'celeryworker'))
+  server_flags_set = args.server or args.command == 'server'
+  worker_flags_set = args.command in ('psqworker', 'celeryworker')
   if args.run_local and (server_flags_set or worker_flags_set):
     log.error('--run_local flag is not compatible with server/worker flags')
     sys.exit(1)
@@ -470,9 +465,8 @@ def main():
           'Creating request {0:s} with evidence {1:s}'.format(
               request.request_id, evidence_.name))
       log.info(
-          'Run command "turbiniactl {0:s} status -r {1:s}" to see the status of'
-          ' this request and associated tasks'.format(
-              '-C' if args.use_celery else '', request.request_id))
+          'Run command "turbiniactl status -r {0:s}" to see the status of'
+          ' this request and associated tasks'.format(request.request_id))
       if not args.run_local:
         client.send_request(request)
       else:


Test patch:
Repository: google/turbinia

Issue:
Local installation requires filled-out GCP section of config
I'm attempting to get Turbinia working locally on a laptop. The laptop has internet for at least the installation part of this process.

Running into an issue with the config. I set the TASK_MANAGER to use Celery, but the installation process is still requiring me to have values for:

- TURBINIA_PROJECT
- TURBINIA_ZONE
- TURBINIA_REGION
- BUCKET_NAME

According to the instructions in the config, these values should only matter if the TASK_MANAGER is set to PSQ.


Discussion:
Hrm, those should not be required variables, so you should be able to set them to None.  What are they set to right now?  See here for what variables are required vs. optional:
https://github.com/google/turbinia/blob/d42bad06b1758a0b89ae873e99a75c09ecdbf0aa/turbinia/config/__init__.py#L37
I'm wondering if it's because I originally had the STATE_MANAGER set to an incorrect value.  Will have to test again later.
Yeah, quite possible.  Right now for local installations, both the STATE_MANAGER and TASK_MANAGER need to be set to redis/celery respectively (and for GCP they should be set to datastore/psq respectively).  In theory, mixing them could work, but for now it's untested and not-recommended.  I'll make a note of this in the docs, and will update the comments in the config file too.

Resolving patch:
diff --git a/turbinia/config/turbinia_config.py b/turbinia/config/turbinia_config.py
--- a/turbinia/config/turbinia_config.py
+++ b/turbinia/config/turbinia_config.py
@@ -26,10 +26,14 @@
 # separate when running with the same Cloud projects or backend servers.
 INSTANCE_ID = 'turbinia-instance1'
 
-# Which state manager to use. Valid options are 'Datastore' or 'Redis'.
+# Which state manager to use. Valid options are 'Datastore' or 'Redis'.  Use
+# 'Datastore' for Cloud (GCP) or hybrid installations, and 'Redis' for local
+# installations.
 STATE_MANAGER = 'Datastore'
 
-# Which Task manager to use. Valid options are 'PSQ' and 'Celery'.
+# Which Task manager to use. Valid options are 'PSQ' and 'Celery'.  Use 'PSQ'
+# for Cloud (GCP) or hybrid installations, and 'Celery' for local
+# installations.
 TASK_MANAGER = 'PSQ'
 
 # Default base output directory for worker results and evidence.


Test patch:
Repository: google/turbinia

Issue:
worker version check failure crashes server
If the worker version doesn't match the server it is supposed to error out and the server should handle that gracefully and report it in the task status.  Right now we get this on the server:
```
Traceback (most recent call last):
  File "/usr/local/bin/turbiniactl", line 11, in <module>
    load_entry_point('turbinia==20181004', 'console_scripts', 'turbiniactl')()
  File "/usr/local/lib/python2.7/dist-packages/turbinia/turbiniactl.py", line 467, in main
    server.start()
  File "/usr/local/lib/python2.7/dist-packages/turbinia/client.py", line 766, in start
    self.task_manager.run()
  File "/usr/local/lib/python2.7/dist-packages/turbinia/task_manager.py", line 281, in run
    for task in self.process_tasks():
  File "/usr/local/lib/python2.7/dist-packages/turbinia/task_manager.py", line 426, in process_tasks
    task.stub.result(timeout=PSQ_TASK_TIMEOUT_SECONDS))
  File "/usr/local/lib/python2.7/dist-packages/turbinia/workers/__init__.py", line 287, in deserialize
    result.__dict__.update(input_dict)
TypeError: 'TurbiniaTaskResult' object is not iterable
```
And the worker:
[ERROR] Worker and Server versions do not match: 20181004 != 20190819

Also need to double check that the post-processors run properly to clean things up.




Discussion:

Resolving patch:
diff --git a/turbinia/workers/__init__.py b/turbinia/workers/__init__.py
--- a/turbinia/workers/__init__.py
+++ b/turbinia/workers/__init__.py
@@ -613,7 +613,7 @@ def run_wrapper(self, evidence):
                   self.turbinia_version, turbinia.__version__))
           self.result.log(message, level=logging.ERROR)
           self.result.status = message
-          return self.result
+          return self.result.serialize()
 
         self._evidence_config = evidence.config
         self.result = self.run(evidence, self.result)


Test patch:
Repository: google/turbinia

Issue:
Provide error message when job whitelist flag has entires that are not valid jobs 
Right now turbiniactl will not validate if a job actually exists. 


Discussion:

Resolving patch:
diff --git a/turbinia/client.py b/turbinia/client.py
--- a/turbinia/client.py
+++ b/turbinia/client.py
@@ -957,7 +957,6 @@ def __init__(self, jobs_blacklist=None, jobs_whitelist=None):
       msg = 'Error creating PSQ Queue: {0:s}'.format(str(e))
       log.error(msg)
       raise TurbiniaException(msg)
-
     # Deregister jobs from blacklist/whitelist.
     disabled_jobs = list(config.DISABLED_JOBS) if config.DISABLED_JOBS else []
     job_manager.JobsManager.DeregisterJobs(jobs_blacklist, jobs_whitelist)
diff --git a/turbinia/jobs/manager.py b/turbinia/jobs/manager.py
--- a/turbinia/jobs/manager.py
+++ b/turbinia/jobs/manager.py
@@ -108,7 +108,6 @@ def DeregisterJobs(cls, jobs_blacklist=None, jobs_whitelist=None):
     """
     registered_jobs = list(cls.GetJobNames())
     jobs_remove = []
-
     # Create a list of jobs to deregister.
     if jobs_whitelist and jobs_blacklist:
       raise TurbiniaException(
@@ -116,6 +115,11 @@ def DeregisterJobs(cls, jobs_blacklist=None, jobs_whitelist=None):
           'time.')
     elif jobs_whitelist:
       jobs_whitelist = [j.lower() for j in jobs_whitelist]
+      for j in jobs_whitelist:
+        if j not in registered_jobs:
+          msg = 'Error whitelisting jobs: Job {0!s} is not found in registered jobs {1!s}.'.format(
+              j, registered_jobs)
+          raise TurbiniaException(msg)
       jobs_remove = [j for j in registered_jobs if j not in jobs_whitelist]
     elif jobs_blacklist:
       jobs_blacklist = [j.lower() for j in jobs_blacklist]
diff --git a/turbinia/task_manager.py b/turbinia/task_manager.py
--- a/turbinia/task_manager.py
+++ b/turbinia/task_manager.py
@@ -131,6 +131,14 @@ def setup(self, jobs_blacklist=None, jobs_whitelist=None, *args, **kwargs):
     self._backend_setup(*args, **kwargs)
     job_names = jobs_manager.JobsManager.GetJobNames()
     if jobs_blacklist or jobs_whitelist:
+      selected_jobs = jobs_blacklist or jobs_whitelist
+      job_names = list(jobs_manager.JobsManager.GetJobNames())
+      for job in selected_jobs:
+        if job.lower() not in job_names:
+          msg = 'Error creating server. Job {0!s} is not found in registered jobs {1!s}.'.format(
+              job, job_names)
+          log.error(msg)
+          raise TurbiniaException(msg)
       log.info(
           'Filtering Jobs with whitelist {0!s} and blacklist {1!s}'.format(
               jobs_whitelist, jobs_blacklist))


Test patch: