Repository: web2py/web2py

Issue:
#968 seems not fixed yet.
`IS_MATCH('hello')(u'\xff')`

Traceback (most recent call last):
...
  File ".../web2py/gluon/validators.py", line 209, in **call**
    match = self.regex.search(str(value))
UnicodeEncodeError: 'ascii' codec can't encode character u'\xff' in position 0: ordinal not in range(128)



Discussion:
#968 

is_unicode is False by default. It works for me with is_unicode=True as below. Does that resolve your issue?

```
IS_MATCH('hello',is_unicode=True)(u'\xff')
# (u'\xff', 'Invalid expression')
```

The is_unicode parameter seems redundant. You have check if the value is unicode, so why you need is_unicode again?  why not fully deprecate it. My test case will crash the program if I am unware of is_unicode.

@zhaoguixu : IS_MATCH is a validator that has to follow a certain behaviour in order to not break backward compatibility. 
Usually validators are run upon form submissions, which come in not as unicode strings, so a default not contemplating unicode is what the framework needs. 
That's why there's a is_unicode argument that defaults to False.
If you're adamant about a regex validator that takes unicode values as default, we can't change the IS_MATCH one but create a new one, also because the is_unicode parameters passes the re.UNICODE flag to the regex itself.
Please report back if you want a new IS_UMATCH validator that only takes unicode as input and passes the re.UNICODE flag to the regex, else I'll close this as it can't be fixed in the original IS_MATCH validator for the reasons explained above.

 Thanks for the reply. The type check `if isinstance(value,unicode):` has fully checked whether the passed `value` is an unicode value. Well, even if for the compatibility that the "is_unicode" parameter is necessary, how about giving a friendly raise when `is_unicode == False` while `isinstance(value, unicode)==True` or totally following the real format of the passed value (ignoring the `is_unicode`)?  Since `is_unicode` is manually set and the user may sometime not recognize the real format of the value and make `is_unicode` as False by default, the unfriendly raise will occur when such a mismatching happened. Feel free to close this if you are not quite into my suggestion.

whoops, I'm lost a taddle bit.
Let's recap:
is_unicode changes the behaviour in two places: 
1. it passes the unicode flag to the regex, "translating" the regex itself
2. it "translates" the passed input to unicode before the actual validation

Now, this two-fold behaviour is the truly responsible for the unfortunate usecase that you reported, but I don't see a way around it because it still has the 1. point to go through.
This validator is also one of the few - unfortunate - cases where the input value is actually changed according to options passed (a str value passed with is_unicode=True gets returned as unicode).

Raising a friendly-error when is_unicode is False but isinstance(value, unicode) == True has the benefit of warning development-mode-users that something is going wrong, but is unprecedented in validators: the only raise statements there if I'm not mistaken are when you set some validators' option to something that won't work in any validation scenario.
In the case you're advocating it would raise an error that is dependent on the validation at runtime, not at "instantiation-time".
Hope you get my POV on this: it's full of intricacies but feel free to ask for more details or explanations, it's totally possible I'm missing a loophole here ^_^

Yes, `is_unicode` is set to be False during the instantiation time while an unicode `value` is passed as the input during the validation time. This will cause the raise as my case. I don't think the check `if self.is_unicode` is necessary during validation time. Hope I have got your point, and feel free to handle this. Thanks.

So, how about this fix?

``` python
def __call__(self, value):
    if self.is_unicode:
        if not isinstance(value, unicode):
            match = self.regex.search(str(value).decode('utf8'))
        else:
            match = self.regex.search(value)
    else:
        if not isinstance(value, unicode):
            match = self.regex.search(str(value))
        else:
            match = self.regex.search(value.encode('utf8')) 
```


Resolving patch:
diff --git a/gluon/validators.py b/gluon/validators.py
--- a/gluon/validators.py
+++ b/gluon/validators.py
@@ -201,12 +201,15 @@ def __init__(self, expression, error_message='Invalid expression',
 
     def __call__(self, value):
         if self.is_unicode:
-            if isinstance(value,unicode):
-                match = self.regex.search(value)
-            else:
+            if not isinstance(value, unicode):
                 match = self.regex.search(str(value).decode('utf8'))
+            else:
+                match = self.regex.search(value)
         else:
-            match = self.regex.search(str(value))
+            if not isinstance(value, unicode):
+                match = self.regex.search(str(value))
+            else:
+                match = self.regex.search(value.encode('utf8'))
         if match is not None:
             return (self.extract and match.group() or value, None)
         return (value, translate(self.error_message))


Test patch:
diff --git a/gluon/tests/test_validators.py b/gluon/tests/test_validators.py
--- a/gluon/tests/test_validators.py
+++ b/gluon/tests/test_validators.py
@@ -618,6 +618,9 @@ def test_IS_MATCH(self):
         self.assertEqual(rtn, (u'hell', None))
         rtn = IS_MATCH('hell', is_unicode=True)(u'hell')
         self.assertEqual(rtn, (u'hell', None))
+        # regr test for #1044
+        rtn = IS_MATCH('hello')(u'\xff')
+        self.assertEqual(rtn, (u'\xff', 'Invalid expression'))
 
 
     def test_IS_EQUAL_TO(self):