anon-iclr-submission/benchname-bug-localization huggingface dataset:
Repository: pantsbuild/pex
Issue: 'pex <dir>' doesn't actually work unless an .egg-info is already there

It turns out that pex for some reason generates an .egg-info missing setup.py, which causes 'pex .' to fail in a fresh clone.

``` bash
mba=flask=; ~/clients/pex/dist/pex -v -v -v -v -v .
**** Failed to install Flask-0.11.dev0. stdout:

**** Failed to install Flask-0.11.dev0. stderr:
Traceback (most recent call last):
  File "<stdin>", line 6, in <module>
IOError: [Errno 2] No such file or directory: 'setup.py'
```


Changed files:
pex/installer.py, pex/testing.py

diff --git a/pex/installer.py b/pex/installer.py
index 38958a6..17f4970 100644
--- a/pex/installer.py
+++ b/pex/installer.py
@@ -35,6 +35,7 @@ class InstallerBase(object):
   SETUP_BOOTSTRAP_MODULE = "sys.path.insert(0, %(path)r); import %(module)s"
   SETUP_BOOTSTRAP_FOOTER = """
 __file__ = 'setup.py'
+sys.argv[0] = 'setup.py'
 exec(compile(open(__file__).read().replace('\\\\r\\\\n', '\\\\n'), __file__, 'exec'))
 """
 
diff --git a/pex/testing.py b/pex/testing.py
index 7d07310..54c14c0 100644
--- a/pex/testing.py
+++ b/pex/testing.py
@@ -79,9 +79,6 @@ PROJECT_CONTENT = {
           package_data={'my_package': ['package_data/*.dat']},
       )
   '''),
-  'MANIFEST.in': dedent('''
-  include setup.py
-  '''),
   'scripts/hello_world': '#!/usr/bin/env python\\nprint("hello world!")\\n',
   'scripts/shell_script': '#!/usr/bin/env bash\\necho hello world\\n',
   'my_package/__init__.py': 0,
Repository: pantsbuild/pex
Issue: ValueError: list.remove(x): x not in list

Something must be broken with PEX_IGNORE_ERRORS

```
$ pex flask -o /tmp/flask.pex
$ PEX_PYTHON=pypy /tmp/flask.pex
Traceback (most recent call last):
  File "/tmp/flask.pex/.bootstrap/_pex/pex.py", line 311, in execute
    working_set = self._activate()
  File "/tmp/flask.pex/.bootstrap/_pex/pex.py", line 76, in _activate
    for dist in env.activate():
  File "/tmp/flask.pex/.bootstrap/_pex/environment.py", line 132, in activate
    self._working_set = self._activate()
  File "/tmp/flask.pex/.bootstrap/_pex/environment.py", line 179, in _activate
    resolved = self._resolve(working_set, all_reqs)
  File "/tmp/flask.pex/.bootstrap/_pex/environment.py", line 150, in _resolve
    reqs.remove(unresolved_req)
ValueError: list.remove(x): x not in list

```


Changed files:
pex/environment.py

diff --git a/pex/environment.py b/pex/environment.py
index c4df78b..efc92d9 100644
--- a/pex/environment.py
+++ b/pex/environment.py
@@ -120,7 +120,7 @@ class PEXEnvironment(Environment):
   def update_candidate_distributions(self, distribution_iter):
     for dist in distribution_iter:
       if self.can_add(dist):
-        with TRACER.timed('Adding %s:%s' % (dist, dist.location), V=2):
+        with TRACER.timed('Adding %s' % dist, V=2):
           self.add(dist)
 
   def can_add(self, dist):
@@ -136,20 +136,22 @@ class PEXEnvironment(Environment):
 
   def _resolve(self, working_set, reqs):
     reqs = reqs[:]
-    unresolved_reqs = []
+    unresolved_reqs = set()
+    resolveds = set()
 
-    while True:
-      with TRACER.timed('Resolving %s' %
-          ' '.join(map(str, reqs)) if reqs else 'empty dependency list', V=2):
+    # Resolve them one at a time so that we can figure out which ones we need to elide should
+    # there be an interpreter incompatibility.
+    for req in reqs:
+      with TRACER.timed('Resolving %s' % req, V=2):
         try:
-          resolved = working_set.resolve(reqs, env=self)
+          resolveds.update(working_set.resolve([req], env=self))
         except DistributionNotFound as e:
           TRACER.log('Failed to resolve a requirement: %s' % e)
-          unresolved_req = e.args[0]
-          unresolved_reqs.append(unresolved_req)
-          reqs.remove(unresolved_req)
-        else:
-          break
+          unresolved_reqs.add(e.args[0].project_name)
+          if e.args[1]:
+            unresolved_reqs.update(e.args[1])
+
+    unresolved_reqs = set([req.lower() for req in unresolved_reqs])
 
     if unresolved_reqs:
       TRACER.log('Unresolved requirements:')
@@ -165,7 +167,7 @@ class PEXEnvironment(Environment):
         die('Failed to execute PEX file, missing compatible dependencies for:\\n%s' % (
             '\\n'.join(map(str, unresolved_reqs))))
 
-    return resolved
+    return resolveds
 
   def _activate(self):
     self.update_candidate_distributions(self.load_internal_cache(self._pex, self._pex_info))
Repository: google/timesketch
Issue: Wrong app context for CSV task

We need to run the CSV importer task in the correct context.

Changed files:
timesketch/lib/tasks.py

diff --git a/timesketch/lib/tasks.py b/timesketch/lib/tasks.py
index 83d0b164..b91cef84 100644
--- a/timesketch/lib/tasks.py
+++ b/timesketch/lib/tasks.py
@@ -121,9 +121,10 @@ def run_csv(source_file_path, timeline_name, index_name, username=None):
     total_events = es.import_event(flush_interval, index_name, event_type)
 
     # We are done so let's remove the processing status flag
-    search_index = SearchIndex.query.filter_by(index_name=index_name).first()
-    search_index.status.remove(search_index.status[0])
-    db_session.add(search_index)
-    db_session.commit()
+    with celery.app.app_context():
+        search_index = SearchIndex.query.filter_by(index_name=index_name).first()
+        search_index.status.remove(search_index.status[0])
+        db_session.add(search_index)
+        db_session.commit()
 
     return {u'Events processed': total_events}
Repository: google/timesketch
Issue: Wrong error message in timesketch_importer

When an upload filed it gives out the timesketch timeline description instead of the status.

https://github.com/google/timesketch/blob/3907a5982bcd2b97ee03cdf816adeddceaead34f/importer_client/python/tools/timesketch_importer.py#L478

Changed files:
importer_client/python/timesketch_import_client/version.py, importer_client/python/tools/timesketch_importer.py

diff --git a/importer_client/python/timesketch_import_client/version.py b/importer_client/python/timesketch_import_client/version.py
index 65fd0e4f..5cf22ef4 100644
--- a/importer_client/python/timesketch_import_client/version.py
+++ b/importer_client/python/timesketch_import_client/version.py
@@ -13,7 +13,7 @@
 # limitations under the License.
 """Version information for Timesketch Import Client."""
 
-__version__ = '20210602'
+__version__ = '20210712'
 
 
 def get_version():
diff --git a/importer_client/python/tools/timesketch_importer.py b/importer_client/python/tools/timesketch_importer.py
index 231b4306..bcbc97f0 100644
--- a/importer_client/python/tools/timesketch_importer.py
+++ b/importer_client/python/tools/timesketch_importer.py
@@ -474,8 +474,8 @@ def main(args=None):
         status = timeline.status
         if status in ('archived', 'failed', 'fail'):
             print('[FAIL]')
-            print('Unable to index timeline, reason: {0:s}'.format(
-                timeline.description))
+            print('Unable to index timeline {0:s}, reason: {1:s}'.format(
+                timeline.description ,timeline.status))
             return
 
         if status not in ('ready', 'success'):
Repository: google/timesketch
Issue: New authentication cannot be performed on terminals with no tty

**Describe the bug**
When authenticating for the first time (or after removing a token) the user is presented with a url to visit via stdout, and a token is entered via stdin. In a session with no, or non default tty's, this means the user will not see the url, and may not be able to enter the token.

**To Reproduce**
Steps to reproduce the behavior:
1. Set up an environment with no standard tty. An example is timesketch api user [DFTimewolf](https://github.com/log2timeline/dftimewolf), which has recently implemented a curses UX. 
2. Remove your timesketch token from `~/.timesketch.token`
3. Run the tool

**Expected behavior**
The user should be notified that the session won't support providing a URL to visit, and cannot accept the required token

**Actual behavior**
The user is not aware they need to reauthenticate, as stdout is not available. A user cannot paste the authentication token into the terminal as stdin is not available. Instead, the application blocks on stdin with no indication of the problem.

Changed files:
api_client/python/timesketch_api_client/client.py

diff --git a/api_client/python/timesketch_api_client/client.py b/api_client/python/timesketch_api_client/client.py
index 734229b6..77fe0e09 100644
--- a/api_client/python/timesketch_api_client/client.py
+++ b/api_client/python/timesketch_api_client/client.py
@@ -16,6 +16,7 @@ from __future__ import unicode_literals
 
 import os
 import logging
+import sys
 
 # pylint: disable=wrong-import-order
 import bs4
@@ -246,6 +247,11 @@ class TimesketchApi:
         if run_server:
             _ = flow.run_local_server()
         else:
+            if not sys.stdout.isatty() or not sys.stdin.isatty():
+                msg = ('You will be asked to paste a token into this session to'
+                    'authenticate, but the session doesn\\'t have a tty')
+                raise RuntimeError(msg)
+
             auth_url, _ = flow.authorization_url(prompt="select_account")
 
             if skip_open:
Repository: google/timesketch
Issue: View is not saved when URL parameters are present

The user view is not saved to the database if the view is created from URL parameters.


Changed files:
timesketch/ui/views/sketch.py

diff --git a/timesketch/ui/views/sketch.py b/timesketch/ui/views/sketch.py
index 375a27cb..6f424140 100644
--- a/timesketch/ui/views/sketch.py
+++ b/timesketch/ui/views/sketch.py
@@ -171,6 +171,8 @@ def explore(sketch_id, view_id=None):
     Returns:
         Template with context.
     """
+    DEFAULT_LIMIT = 100  # Number of events to return.
+    save_view = False  # If the view should be saved to the database.
     sketch = Sketch.query.get_with_acl(sketch_id)
     sketch_timelines = [t.searchindex.index_name for t in sketch.timelines]
     view_form = SaveViewForm()
@@ -196,12 +198,11 @@ def explore(sketch_id, view_id=None):
         view = sketch.get_user_view(current_user)
 
     if not view:
-        query_filter = dict(indices=sketch_timelines)
+        query_filter = dict(indices=sketch_timelines, limit=DEFAULT_LIMIT)
         view = View(
-            user=current_user, name=u'', sketch=sketch, query_string=u'',
+            user=current_user, name=u'', sketch=sketch, query_string=u'*',
             query_filter=json.dumps(query_filter, ensure_ascii=False))
-        db_session.add(view)
-        db_session.commit()
+        save_view = True
 
     if url_query:
         view.query_string = url_query
@@ -213,6 +214,17 @@ def explore(sketch_id, view_id=None):
         if url_limit:
             query_filter[u'limit'] = url_limit
         view.query_filter = json.dumps(query_filter, ensure_ascii=False)
+        save_view = True
+
+    query_filter = json.loads(view.query_filter)
+    if not query_filter.get(u'limit', None):
+        query_filter[u'limit'] = DEFAULT_LIMIT
+        view.query_filter = json.dumps(query_filter, ensure_ascii=False)
+        save_view = True
+
+    if save_view:
+        db_session.add(view)
+        db_session.commit()
 
     return render_template(
         u'sketch/explore.html', sketch=sketch, view=view, named_view=view_id,
Repository: google/timesketch
Issue: Unable to get event and add event annotation when using PGSQL

When using postgresql some queries fails with DataError.

This is due to the way SQLalchemy constructs the query, and the way we pass in the searchindex name to the filter.


Changed files:
timesketch/api/v1/resources.py, wsgi.py, timesketch/api/v1/resources_test.py

diff --git a/timesketch/api/v1/resources.py b/timesketch/api/v1/resources.py
index 8e8822e0..6b1aa8ec 100644
--- a/timesketch/api/v1/resources.py
+++ b/timesketch/api/v1/resources.py
@@ -374,7 +374,8 @@ class EventResource(ResourceMixin, Resource):
         args = self.parser.parse_args()
         sketch = Sketch.query.get_with_acl(sketch_id)
         searchindex_id = args.get(u'searchindex_id')
-        searchindex = SearchIndex.query.get(searchindex_id)
+        searchindex = SearchIndex.query.filter_by(
+            index_name=searchindex_id).first()
         event_id = args.get(u'event_id')
         indices = [t.searchindex.index_name for t in sketch.timelines]
 
@@ -428,7 +429,8 @@ class EventAnnotationResource(ResourceMixin, Resource):
             indices = [t.searchindex.index_name for t in sketch.timelines]
             annotation_type = form.annotation_type.data
             searchindex_id = form.searchindex_id.data
-            searchindex = SearchIndex.query.get(searchindex_id)
+            searchindex = SearchIndex.query.filter_by(
+                index_name=searchindex_id).first()
             event_id = form.event_id.data
             event_type = form.event_type.data
 
diff --git a/timesketch/api/v1/resources_test.py b/timesketch/api/v1/resources_test.py
index 954f3d77..6d2f14a2 100644
--- a/timesketch/api/v1/resources_test.py
+++ b/timesketch/api/v1/resources_test.py
@@ -149,9 +149,6 @@ class EventResourceTest(BaseTest):
     """Test EventResource."""
     resource_url = u'/api/v1/sketches/1/event/'
     expected_response = {
-        u'meta': {
-            u'comments': []
-        },
         u'objects': {
             u'timestamp_desc': u'',
             u'timestamp': 1410895419859714,
@@ -172,7 +169,7 @@ class EventResourceTest(BaseTest):
         self.login()
         response = self.client.get(
             self.resource_url + u'?searchindex_id=test&event_id=test')
-        self.assertDictEqual(response.json, self.expected_response)
+        self.assertDictContainsSubset(self.expected_response, response.json)
         self.assert200(response)
 
     @mock.patch(
diff --git a/wsgi.py b/wsgi.py
index 896b1b6e..996e578b 100644
--- a/wsgi.py
+++ b/wsgi.py
@@ -33,5 +33,11 @@ Example configuration for Apache with mod_wsgi (a2enmod mod_wsgi):
 #execfile(activate_virtualenv, dict(__file__=activate_virtualenv))
 
 from timesketch import create_app
+from timesketch.models import db_session
 
 application = create_app()
+
+# Remove the session after every request or app shutdown.
+@application.teardown_appcontext
+def shutdown_session(exception=None):
+    db_session.remove()
Repository: google/timesketch
Issue: tagger analyzer not functiong properly 

**Describe the bug**
After upgrade TimeSketch to version: 20210602 the tagger analyzer is not functioning with custom tags

**To Reproduce**
Steps to reproduce the behavior:
1. Import plaso file with evtx data 
2. Add the following tagging rule to tags.yaml
```yaml
logon_tagger:  
  query_string: 'data_type: "windows:evtx:record" AND source_name: "Microsoft-Windows-Security-Auditing" AND event_identifier: 4688'
  tags: ['logon']
  save_search: true
  search_name: 'logon'
```
3. run tagger analyzer
4. See error

**Expected behavior**
The tagger analyzer to run correctly as in previous versions.

**Desktop (please complete the following information):**
-OS:Ubuntu 20.04.2 LTS
-Browser : Firefox
-Version: 86.0

**Additional context**
The following exception is thrown once the tagger analyzer is ran:
```
Traceback (most recent call last): File "/usr/local/lib/python3.8/dist-packages/timesketch/lib/analyzers/interface.py", line 995, in run_wrapper result = self.run() File "/usr/local/lib/python3.8/dist-packages/timesketch/lib/analyzers/tagger.py", line 48, in run tag_result = self.tagger(name, tag_config) File "/usr/local/lib/python3.8/dist-packages/timesketch/lib/analyzers/tagger.py", line 100, in tagger if expression: UnboundLocalError: local variable 'expression' referenced before assignment
``` 


Changed files:
timesketch/lib/analyzers/tagger.py

diff --git a/timesketch/lib/analyzers/tagger.py b/timesketch/lib/analyzers/tagger.py
index 0335adf6..712489b9 100644
--- a/timesketch/lib/analyzers/tagger.py
+++ b/timesketch/lib/analyzers/tagger.py
@@ -83,6 +83,7 @@ class TaggerSketchPlugin(interface.BaseAnalyzer):
 
         expression_string = config.get('regular_expression', '')
         attributes = None
+        expression = None
         if expression_string:
             expression = utils.compile_regular_expression(
                 expression_string=expression_string,