ENH: Set environment in try/finally

To avoid situations in wich the flag is not unset

@ogrisel: can I haz review?