internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: web2py/web2py

Issue:
py3: Error importing csv file (using appadmin and in code) 
When trying to import a csv file in appadmin, I get a flash message:
```
iterator should return strings, not bytes (did you open the file in text mode?)
```

I tried to import using code:
```
db.mytable.import_from_csv_file(open('db_mytable.csv', 'r'))
```

and got exactly the same message: 

```
<class '_csv.Error'> iterator should return strings, not bytes (did you open the file in text mode?)
```
I got it working in my code with 
```
db.mytable.import_from_csv_file(open('db_mytable.csv', 'r',encoding='utf-8'))
```
Looks for it works with python3, needs to be explicit declared the encoding. 




Discussion:
Thanks for reporting, as you proposed such issue can be fixed by importing
```from gluon.fileutils import open_file```
and then, replacing 
```db.mytable.import_from_csv_file(open('db_mytable.csv', 'r'))```
with
```db.mytable.import_from_csv_file(open_file('db_mytable.csv', 'r'))```

Would you post a pull request with such fix?

Resolving patch:
diff --git a/scripts/cpdb.py b/scripts/cpdb.py
--- a/scripts/cpdb.py
+++ b/scripts/cpdb.py
@@ -15,6 +15,7 @@
   import readline
 try:
     from gluon import DAL
+    from gluon.fileutils import open_file
 except ImportError as err:
     print('gluon path not found')
 
@@ -531,7 +532,7 @@ def copyDB(self):
         self.db.export_to_csv_file(open('tmp.sql', 'wb'))
 
         print 'importing data...'
-        other_db.import_from_csv_file(open('tmp.sql', 'rb'))
+        other_db.import_from_csv_file(open_file('tmp.sql', 'rb'))
         other_db.commit()
         print 'done!'
         print 'Attention: do not run this program again or you end up with duplicate records'


Test patch:
Repository: web2py/web2py

Issue:
Unicode in XML sanitizing causes error
If i pass a unicode string to the `XML` helper with `sanitize=True`, it throws an error:

```python
In [39]: XML(u'<div>è</div>', sanitize=True)
---------------------------------------------------------------------------
UnicodeEncodeError                        Traceback (most recent call last)
----> 1 XML(u'<div>è</div>', sanitize=True)

./web2py/gluon/html.pyc in __init__(self, text, sanitize, permitted_tags, allowed_attributes)
    607 
    608         if sanitize:
--> 609             text = sanitizer.sanitize(text, permitted_tags, allowed_attributes)
    610         if isinstance(text, unicode):
    611             text = text.encode('utf8', 'xmlcharrefreplace')

./web2py/gluon/sanitizer.pyc in sanitize(text, permitted_tags, allowed_attributes, escape)
    219         return str(text)
    220     return XssCleaner(permitted_tags=permitted_tags,
--> 221                       allowed_attributes=allowed_attributes).strip(text, escape)

./web2py/gluon/sanitizer.pyc in strip(self, rawstring, escape)
    164 
    165         if not isinstance(rawstring, str):
--> 166             return str(rawstring)
    167         for tag in self.requires_no_close:
    168             rawstring = rawstring.replace("<%s/>" % tag, "<%s />" % tag)

UnicodeEncodeError: 'ascii' codec can't encode character u'\xe8' in position 5: ordinal not in range(128)
```


Discussion:

Resolving patch:
diff --git a/gluon/html.py b/gluon/html.py
--- a/gluon/html.py
+++ b/gluon/html.py
@@ -596,10 +596,10 @@ def __init__(
                 for A, IMG and BlockQuote).
                 The key is the tag; the value is a list of allowed attributes.
         """
-        if sanitize:
-            text = sanitizer.sanitize(text, permitted_tags, allowed_attributes)
         if isinstance(text, unicodeT):
             text = to_native(text.encode('utf8', 'xmlcharrefreplace'))
+        if sanitize:
+            text = sanitizer.sanitize(text, permitted_tags, allowed_attributes)
         elif isinstance(text, bytes):
             text = to_native(text)
         elif not isinstance(text, str):


Test patch:
diff --git a/gluon/tests/test_html.py b/gluon/tests/test_html.py
--- a/gluon/tests/test_html.py
+++ b/gluon/tests/test_html.py
@@ -168,6 +168,8 @@ def test_XML(self):
         # seams that __repr__ is no longer enough
         ##self.assertEqual(XML('1.3'), '1.3')
         self.assertEqual(XML(u'<div>è</div>').xml(), b'<div>\xc3\xa8</div>')
+        # make sure unicode works with sanitize
+        self.assertEqual(XML(u'<div>è</div>', sanitize=True).xml(), b'<div>\xc3\xa8</div>')
         # you can calc len on the class, that equals the xml() and the str()
         ##self.assertEqual(len(XML('1.3')), len('1.3'))
         self.assertEqual(len(XML('1.3').xml()), len('1.3'))
Repository: web2py/web2py

Issue:
IS_IN_DB breaks with None values and id/integer fields
If the `field` argument to `IS_IN_DB` is an `id` or `integer` field (which is the case for all reference fields) and the value submitted for validation is `None` (or anything other than an integer or a digit string), an exception is thrown at [this line](https://github.com/web2py/web2py/blob/8a741023d8a2631a0a33dc47ee08f73e71018efc/gluon/validators.py#L635) or [this line](https://github.com/web2py/web2py/blob/8a741023d8a2631a0a33dc47ee08f73e71018efc/gluon/validators.py#L662).

We could probably change:

    if isinstance(value, (int, long)) or value.isdigit():

to:

    if isinstance(value, (int, long)) or (hasattr(value, 'isdigit') and value.isdigit()):



Discussion:

Resolving patch:
diff --git a/gluon/validators.py b/gluon/validators.py
--- a/gluon/validators.py
+++ b/gluon/validators.py
@@ -21,7 +21,7 @@
 import struct
 import decimal
 import unicodedata
-from gluon._compat import StringIO, long, basestring, unicodeT, to_unicode, urllib_unquote, unichr, to_bytes, PY2, to_unicode, to_native
+from gluon._compat import StringIO, long, basestring, unicodeT, to_unicode, urllib_unquote, unichr, to_bytes, PY2, to_unicode, to_native, string_types
 from gluon.utils import simple_hash, web2py_uuid, DIGEST_ALG_BY_SIZE
 from pydal.objects import Field, FieldVirtual, FieldMethod
 from functools import reduce
@@ -659,7 +659,7 @@ def count(values, s=self.dbset, f=field):
                     return (values, None)
         else:
             if field.type in ('id', 'integer'):
-                if isinstance(value, (int, long)) or value.isdigit():
+                if isinstance(value, (int, long)) or (isinstance(value, string_types) and value.isdigit()):
                     value = int(value)
                 elif self.auto_add:
                     value = self.maybe_add(table, self.fieldnames[0], value)


Test patch:
diff --git a/gluon/tests/test_validators.py b/gluon/tests/test_validators.py
--- a/gluon/tests/test_validators.py
+++ b/gluon/tests/test_validators.py
@@ -235,6 +235,11 @@ def test_IS_IN_DB(self):
         self.assertEqual(sorted(rtn), [('%d' % george_id, 'george'), ('%d' % costanza_id, 'costanza')])
         rtn = IS_IN_DB(db, db.person.id, db.person.name, error_message='oops', sort=True).options(zero=True)
         self.assertEqual(rtn, [('', ''), ('%d' % costanza_id, 'costanza'), ('%d' % george_id, 'george')])
+        # Test None
+        rtn = IS_IN_DB(db, 'person.id', '%(name)s', error_message='oops')(None)
+        self.assertEqual(rtn, (None, 'oops'))
+        rtn = IS_IN_DB(db, 'person.name', '%(name)s', error_message='oops')(None)
+        self.assertEqual(rtn, (None, 'oops'))
         # Test using the set it made for options
         vldtr = IS_IN_DB(db, 'person.name', '%(name)s', error_message='oops')
         vldtr.options()
Repository: web2py/web2py

Issue:
remain py2.6 stuff that can be delete
Here :
https://github.com/web2py/web2py/blob/0d646fa5e7c731cb5c392adf6a885351e77e4903/gluon/import_all.py#L86

py26_deprecated is used before been declared...

Since we drop py2.6 : https://groups.google.com/d/msg/web2py-developers/xz5o_CF4GOo/ZJm0HImTAAAJ

Shall we delete those line??

I send a PR if you take it...


Discussion:

Resolving patch:
diff --git a/gluon/import_all.py b/gluon/import_all.py
--- a/gluon/import_all.py
+++ b/gluon/import_all.py
@@ -78,13 +78,9 @@
 # Now we remove the blacklisted modules if we are using the stated
 # python version.
 #
-# List of modules deprecated in Python 2.6 or 2.7 that are in the above set
+# List of modules deprecated in Python 2.7 that are in the above list
 py27_deprecated = ['mhlib', 'multifile', 'mimify', 'sets', 'MimeWriter']  # And ['optparse'] but we need it for now
 
-if python_version >= '2.6':
-    base_modules += ['json', 'multiprocessing']
-    base_modules = list(set(base_modules).difference(set(py26_deprecated)))
-
 if python_version >= '2.7':
     base_modules += ['argparse', 'json', 'multiprocessing']
     base_modules = list(set(base_modules).difference(set(py27_deprecated)))


Test patch:
Repository: web2py/web2py

Issue:
heroku ADAPTERS error
I'm looking to use Heroku for deployment of my web2py project. However, when I add 
`from gluon.contrib.heroku import get_db`
`db = get_db(name=None, pool_size=myconf.get('db.pool_size'))`

I get a ticket with the error:

> File "/Users/huangyu/dev/web2py/gluon/contrib/heroku.py", line 10, in <module>
    from pydal.adapters import ADAPTERS, PostgreSQLAdapter
ImportError: cannot import name ADAPTERS

It looks like web2py has moved on from using ADAPTERS? Has that been replaced by 
`@adapters.register_for('postgres')`

But the heroku file has not been updated. 


Discussion:

Resolving patch:
diff --git a/gluon/contrib/heroku.py b/gluon/contrib/heroku.py
--- a/gluon/contrib/heroku.py
+++ b/gluon/contrib/heroku.py
@@ -7,15 +7,13 @@
 """
 import os
 from gluon import *
-from pydal.adapters import ADAPTERS, PostgreSQLAdapter
-from pydal.helpers.classes import UseDatabaseStoredFile
+from pydal.adapters import adapters, PostgrePsyco
+from pydal.helpers.classes import DatabaseStoredFile
 
-class HerokuPostgresAdapter(UseDatabaseStoredFile,PostgreSQLAdapter):
-    drivers = ('psycopg2',)
+@adapters.register_for('postgres')
+class HerokuPostgresAdapter(DatabaseStoredFile, PostgrePsyco):
     uploads_in_blob = True
 
-ADAPTERS['postgres'] = HerokuPostgresAdapter
-
 def get_db(name = None, pool_size=10):
     if not name:
         names = [n for n in os.environ.keys()


Test patch:
Repository: web2py/web2py

Issue:
Passing tables to SQLFORM.factory() fails
Example:
`form = SQLFORM.factory(idb.catering, idb.catering_diet)`

File "/home/www-data/internal-apps/gluon/sqlhtml.py", line 1922, in factory
    return SQLFORM(DAL(None).define_table(table_name, *[field.clone() for field in fields]),
AttributeError: 'Table' object has no attribute 'clone'

Since .clone() was added to the list comprehension, it tries to clone tables as well, rather than passing them on to define_table() as is.
Passing tables to SQLFORM.factory() fails
Example:
`form = SQLFORM.factory(idb.catering, idb.catering_diet)`

File "/home/www-data/internal-apps/gluon/sqlhtml.py", line 1922, in factory
    return SQLFORM(DAL(None).define_table(table_name, *[field.clone() for field in fields]),
AttributeError: 'Table' object has no attribute 'clone'

Since .clone() was added to the list comprehension, it tries to clone tables as well, rather than passing them on to define_table() as is.


Discussion:

Resolving patch:
diff --git a/gluon/sqlhtml.py b/gluon/sqlhtml.py
--- a/gluon/sqlhtml.py
+++ b/gluon/sqlhtml.py
@@ -1919,8 +1919,10 @@ def factory(*fields, **attributes):
         if 'table_name' in attributes:
             del attributes['table_name']
 
-        return SQLFORM(DAL(None).define_table(table_name, *[field.clone() for field in fields]),
-                       **attributes)
+        # Clone fields, while passing tables straight through
+        fields_with_clones = [f.clone() if isinstance(f, Field) else f for f in fields]                  
+            
+        return SQLFORM(DAL(None).define_table(table_name, *fields_with_clones), **attributes)
 
     @staticmethod
     def build_query(fields, keywords):


Test patch:
Repository: web2py/web2py

Issue:
Error with autocomplete in version 2.15.2
Versión: web2py.exe para windows 2.15.2-stable+timestamp.2017.07.19.12.18.41
Update all js in the target application with 2.15.2 of welcome
Tested in browsers: Firefox and Chrome.

When entering data in the autocomplete field, the candidate values are not displayed.
When you enter, it indicates an invalid value.
And an image in transparent format appears and disappears from the first part of the displayed panel.
The panel, is to insert record, invoked by the (+) button of a sqlform.grid.
It worked correctly in version 2.14.6 and 2.15.1
This problem occurs in a panel that has been running for months, and has not had updates.

Regards.


Discussion:
I changed autocomplete due to a security issue with the previous version. In my tests it worked fine using appadmin which does use the grid. Can you give me a sample application?
Hi Leonel
Sorry for the delay.
I send imagen panel with problem, noting also, superimposed image and programs.

ER diagram:    db.trabajadores<--->db.trbfam<--->db.familares

![panel](https://user-images.githubusercontent.com/30487405/28694969-ccd9a8f2-7302-11e7-8555-74d24b89a4e1.jpg)

[objetos.zip](https://github.com/web2py/web2py/files/1181436/objetos.zip)


@domezzzzx Thanks, looking at your sample code, I managed to reproduce the bug which happens when user_signature is True on the grid. I'm investigating. BTW this working before is an example of the security flaw which was fixed.

Resolving patch:
diff --git a/gluon/sqlhtml.py b/gluon/sqlhtml.py
--- a/gluon/sqlhtml.py
+++ b/gluon/sqlhtml.py
@@ -658,7 +658,8 @@ def __init__(self, request, field, id_field=None, db=None,
                  orderby=None, limitby=(0, 10), distinct=False,
                  keyword='_autocomplete_%(tablename)s_%(fieldname)s',
                  min_length=2, help_fields=None, help_string=None,
-                 at_beginning=True, default_var='ac'):
+                 at_beginning=True, default_var='ac', user_signature=True,
+                 hash_vars=False):
 
         self.help_fields = help_fields or []
         self.help_string = help_string
@@ -683,7 +684,8 @@ def __init__(self, request, field, id_field=None, db=None,
         if hasattr(request, 'application'):
             urlvars = request.vars
             urlvars[default_var] = 1
-            self.url = URL(args=request.args, vars=urlvars)
+            self.url = URL(args=request.args, vars=urlvars,
+                           user_signature=user_signature, hash_vars=hash_vars)
             self.run_callback = True
         else:
             self.url = request


Test patch:
Repository: web2py/web2py

Issue:
Can't download file with non ascii characters in the filename
Steps to reproduce:
1 - upload file with non ascii characters in the name
2 - try to download it
  
File "C:\Users\Leonel\Desktop\web2py\gluon\main.py", line 564, in wsgibase
    return http_response.to(responder, env=env)

  File "C:\Users\Leonel\Desktop\web2py\gluon\http.py", line 120, in to
    rheaders.append((k, str(v)))

UnicodeEncodeError: 'ascii' codec can't encode character u'\xe2' in position 33: ordinal not in range(128)

Example of a header causing this problem:
Content-Disposition attachment; filename="comacentoscâmara.jpg"

I plan to fix this issue.


Discussion:

Resolving patch:
diff --git a/gluon/globals.py b/gluon/globals.py
--- a/gluon/globals.py
+++ b/gluon/globals.py
@@ -14,7 +14,7 @@
 
 """
 from gluon._compat import pickle, StringIO, copyreg, Cookie, urlparse, PY2, iteritems, to_unicode, to_native, \
-    unicodeT, long, hashlib_md5
+    unicodeT, long, hashlib_md5, urllib_quote
 from gluon.storage import Storage, List
 from gluon.streamer import streamer, stream_file_or_304_or_206, DEFAULT_CHUNK_SIZE
 from gluon.contenttype import contenttype
@@ -641,6 +641,11 @@ def download():
         if download_filename is None:
             download_filename = filename
         if attachment:
+            # Browsers still don't have a simple uniform way to have non ascii
+            # characters in the filename so for now we are percent encoding it
+            if isinstance(download_filename, unicodeT):
+                download_filename = download_filename.encode('utf-8')
+            download_filename = urllib_quote(download_filename)
             headers['Content-Disposition'] = \
                 'attachment; filename="%s"' % download_filename.replace('"', '\"')
         return self.stream(stream, chunk_size=chunk_size, request=request)


Test patch:
Repository: web2py/web2py

Issue:
@Mobilize support missing in 2.15.2?
After upgrading to 2.15.2 views using **mobilize** statement throw an exception.
Before (2.14.6) same views worked fine

I can reproduce this using a minimal example 

from gluon.contrib.user_agent_parser import mobilize
@mobilize
def index():
   return dict()

throws an error
<type 'exceptions.ImportError'> cannot import name mobilize


Traceback (most recent call last):

File "/home/www-data/web2py/gluon/restricted.py", line 219, in restricted
exec(ccode, environment)
File "/home/www-data/web2py/applications/myapp/controllers/stats.py", line 3, in <module>    from gluon.contrib.user_agent_parser import mobilize

ImportError: cannot import name mobilize

Any ideas?


Discussion:
Cross ref to mailing list, a least another user confirms same issue

https://groups.google.com/d/msg/web2py/ooZKk5i8zew/2TckkMq6CAAJ

Found it: mobilize was removed here:

https://github.com/web2py/web2py/commit/d9c7953147bd78bd2e3ebf546b989538ae983c8e#diff-795b1930ce8fe2b530ad43ece7532341L557

but this breaks backward compatibility and is not yet reflected in the documentation (see http://web2py.com/books/default/chapter/29/05/the-views#Mobile-development)

Resolving patch:
diff --git a/gluon/contrib/user_agent_parser.py b/gluon/contrib/user_agent_parser.py
--- a/gluon/contrib/user_agent_parser.py
+++ b/gluon/contrib/user_agent_parser.py
@@ -673,3 +673,23 @@ def simple_detect(agent):
     if os_version:
         os = " ".join((os, os_version))
     return os, browser
+
+
+class mobilize(object):  
+    """
+    Decorator for controller functions so they use different views for mobile devices.
+
+    WARNING: If you update httpagentparser make sure to leave mobilize for
+    backwards compatibility.
+    """
+    def __init__(self, func):
+        self.func = func
+    
+    def __call__(self):
+        from gluon import current
+        user_agent = current.request.user_agent()
+        if user_agent.is_mobile:
+            items = current.response.view.split('.')
+            items.insert(-1, 'mobile')
+            current.response.view = '.'.join(items)
+        return self.func()
diff --git a/gluon/globals.py b/gluon/globals.py
--- a/gluon/globals.py
+++ b/gluon/globals.py
@@ -331,11 +331,16 @@ def user_agent(self):
         user_agent = session._user_agent
         if user_agent:
             return user_agent
-        user_agent = user_agent_parser.detect(self.env.http_user_agent)
+        http_user_agent = self.env.http_user_agent
+        user_agent = user_agent_parser.detect(http_user_agent)
         for key, value in user_agent.items():
             if isinstance(value, dict):
                 user_agent[key] = Storage(value)
-        user_agent = session._user_agent = Storage(user_agent)
+        user_agent = Storage(user_agent)
+        user_agent.is_mobile = 'Mobile' in http_user_agent
+        user_agent.is_tablet = 'Tablet' in http_user_agent
+        session._user_agent = user_agent 
+        
         return user_agent
 
     def requires_https(self):


Test patch:
Repository: web2py/web2py

Issue:
XML sanitize=True throws error on check for acceptable url in py2
Below code in view throws error
```
{{=XML('<img src="/myapp/static/images/pict.jpg" />', sanitize=True)}}
```

Error:
```
Traceback (most recent call last):
  File "\web2py\gluon\restricted.py", line 219, in restricted
    exec(ccode, environment)
  File "<string>", line 73, in <module>
  File "\web2py\gluon\html.py", line 603, in __init__
    text = sanitizer.sanitize(text, permitted_tags, allowed_attributes)
  File "\web2py\gluon\sanitizer.py", line 219, in sanitize
    allowed_attributes=allowed_attributes).strip(text, escape)
  File "\web2py\gluon\sanitizer.py", line 170, in strip
    self.feed(rawstring)
  File "HTMLParser.py", line 117, in feed
  File "HTMLParser.py", line 161, in goahead
  File "HTMLParser.py", line 325, in parse_starttag
  File "HTMLParser.py", line 407, in handle_startendtag
  File "\web2py\gluon\sanitizer.py", line 115, in handle_starttag
    if self.url_is_acceptable(attrs[attribute]):
  File "\web2py\gluon\sanitizer.py", line 148, in url_is_acceptable
    parsed = urlparse(url)
TypeError: 'module' object is not callable
```

Environment info:
- platform: win32
- python: 2.7.9 (major=2, minor=7, micro=9, releaselevel='final', serial=0)
- web2py: 2.15.2-stable+timestamp.2017.07.19.12.18.41

Steps to reproduce:
1. Open web2py from console
2. Try execute: `XML('<img src="/myapp/static/images/pict.jpg" />', sanitize=True)`

Possible fix:
In gluon._compat.py change line 14 from:
`import urlparse`
to 
`from urlparse import urlparse`


Discussion:
Don't change _compat, instead change urlparse in urlparse.urlparse in sanitizer.py, would you post a PR?
hm, but for py3 (else block in _compat) actually a function is imported:
`from urllib import parse as urlparse`
Given this, won't urlparse.urlparse in sanitizer.py throw an error in case of py3?
@lkrn if you pay close attention that's not a function :) that's the parse module in urllib.

Resolving patch:
diff --git a/gluon/sanitizer.py b/gluon/sanitizer.py
--- a/gluon/sanitizer.py
+++ b/gluon/sanitizer.py
@@ -145,7 +145,7 @@ def url_is_acceptable(self, url):
         if url.startswith('#'):
             return True
         else:
-            parsed = urlparse(url)
+            parsed = urlparse.urlparse(url)
             return ((parsed[0] in self.allowed_schemes and '.' in parsed[1]) or
                     (parsed[0] in self.allowed_schemes and '@' in parsed[2]) or
                     (parsed[0] == '' and parsed[2].startswith('/')))


Test patch:
Repository: web2py/web2py

Issue:
python 3 web2py2.15.3 urllib has no getproxies
ben@x1:/home/www-data/web2py$ python3 /home/www-data/web2py/web2py.py -K app-name,app-name

web2py Web Framework
Created by Massimo Di Pierro, Copyright 2007-2017
Version 2.15.3-stable+timestamp.2017.08.07.07.32.04
Database drivers available: pymysql, imaplib, psycopg2, sqlite3, pg8000
Traceback (most recent call last):
  File "/home/www-data/web2py/web2py.py", line 33, in <module>
    gluon.widget.start(cron=True)
  File "/home/www-data/web2py/gluon/widget.py", line 1170, in start
    start_schedulers(options)
  File "/home/www-data/web2py/gluon/widget.py", line 1080, in start_schedulers
    urllib.getproxies()
AttributeError: module 'urllib' has no attribute 'getproxies'



Discussion:

Resolving patch:
diff --git a/gluon/widget.py b/gluon/widget.py
--- a/gluon/widget.py
+++ b/gluon/widget.py
@@ -1076,7 +1076,10 @@ def start_schedulers(options):
         return
 
     # Work around OS X problem: http://bugs.python.org/issue9405
-    import urllib
+    if PY2:
+        import urllib
+    else:
+        import urllib.request as urllib
     urllib.getproxies()
 
     for app in apps:


Test patch: