internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: OCHA-DAP/hdx-ckan

Issue:
Colombia Page:  Data link should not appear if source_link is null
![image](https://cloud.githubusercontent.com/assets/1654485/5373114/8a1f4c6a-8052-11e4-9808-465683e3e231.png)



Discussion:

Resolving patch:
diff --git a/ckanext-hdx_crisis/ckanext/hdx_crisis/dao/data_access.py b/ckanext-hdx_crisis/ckanext/hdx_crisis/dao/data_access.py
--- a/ckanext-hdx_crisis/ckanext/hdx_crisis/dao/data_access.py
+++ b/ckanext-hdx_crisis/ckanext/hdx_crisis/dao/data_access.py
@@ -100,6 +100,10 @@ def fetch_data(self, context):
             item[CrisisDataAccess.UNIQUE_ID_COL]: item for item in self.results
             if CrisisDataAccess.UNIQUE_ID_COL in item}
 
+        for it in self.results:
+            if 'source_link' in it and it['source_link']:
+                it['source_link'] = it['source_link'].strip()
+
         for code, res_dict in self.resources_dict.iteritems():
             if code != 'top-line-numbers':
                 log.info("Fetching data for dataset:{} and resource: {} ".format(


Test patch:
Repository: OCHA-DAP/hdx-ckan

Issue:
Server Error when downloading dataset
This dataset, recently added by @JavierTeran , has been throwing a server error when I click on "download": https://data.hdx.rwlabs.org/dataset/victimas-de-minas/resource/0f39852c-e4bd-4c41-b5a1-88f38564955f



Discussion:
Checking it


Resolving patch:
diff --git a/ckanext-hdx_package/ckanext/hdx_package/controllers/storage_controller.py b/ckanext-hdx_package/ckanext/hdx_package/controllers/storage_controller.py
--- a/ckanext-hdx_package/ckanext/hdx_package/controllers/storage_controller.py
+++ b/ckanext-hdx_package/ckanext/hdx_package/controllers/storage_controller.py
@@ -107,8 +107,9 @@ def _download_file(self, res, label):
                 'Content-Type': metadata.get('_format', 'text/plain')}
             if resource.name:
                 res_name = resource.name.replace('"', '_')
+                res_name_encoded = res_name.encode('utf-8', 'ignore')
                 headers[
-                    'Content-Disposition'] = 'inline; filename="{}"'.format(res_name)
+                    'Content-Disposition'] = 'inline; filename="{}"'.format(res_name_encoded)
             fapp = FileApp(filepath, headers=None, **headers)
             return fapp(request.environ, self.start_response)
         else:


Test patch:
Repository: OCHA-DAP/hdx-ckan

Issue:
Title of the Colombia page should be "Colombia Country Page" or alike
Right now the title of the [Colombia country page](https://data.hdx.rwlabs.org/group/col) is "Colombia crisis page". I think it should read "Colombia Country Page" or similar. Any ideas? 

![screen shot 2014-12-11 at 12 39 13 pm](https://cloud.githubusercontent.com/assets/953118/5398732/e5a76cee-8132-11e4-9f55-900e1c156f6a.png)



Discussion:
Good catch, Luis. It should match our other country pages:  `Colombia - Humanitarian Data Exchange`

Awesome. ::-)


Resolving patch:
diff --git a/ckanext-hdx_theme/ckanext/hdx_theme/version.py b/ckanext-hdx_theme/ckanext/hdx_theme/version.py
--- a/ckanext-hdx_theme/ckanext/hdx_theme/version.py
+++ b/ckanext-hdx_theme/ckanext/hdx_theme/version.py
@@ -1 +1 @@
-hdx_version = 'v0.5.10'
+hdx_version = 'v0.5.11'


Test patch:
Repository: OCHA-DAP/hdx-ckan

Issue:
Member List: An old friend has come back to visit
Also, the sort order has no effect.

![image](https://cloud.githubusercontent.com/assets/1654485/5455616/4e71db8e-853d-11e4-91df-2f20de640400.png)



Discussion:

Resolving patch:
diff --git a/ckanext-hdx_org_group/ckanext/hdx_org_group/controllers/member_controller.py b/ckanext-hdx_org_group/ckanext/hdx_org_group/controllers/member_controller.py
--- a/ckanext-hdx_org_group/ckanext/hdx_org_group/controllers/member_controller.py
+++ b/ckanext-hdx_org_group/ckanext/hdx_org_group/controllers/member_controller.py
@@ -32,7 +32,7 @@ def members(self, id):
 
         q = c.q = request.params.get('q', '')
         sort = request.params.get('sort', '')
-        reverse = True if sort == u'name desc' else False
+        reverse = True if sort == u'title desc' else False
 
         c.sort_by_selected = sort
 
@@ -107,7 +107,7 @@ def member_new(self, id):
         except ValidationError, e:
             h.flash_error(e.error_summary)
         self._redirect_to(controller='group', action='members', id=id)
-        
+
     def member_delete(self, id):
         ''' This is a modified version of the member_delete from the 
             ckan group controller. 
@@ -129,8 +129,10 @@ def member_delete(self, id):
         try:
             user_id = request.params.get('user')
             if request.method == 'POST':
-                self._action('group_member_delete')(context, {'id': id, 'user_id': user_id})
-                h.flash_notice(_('Organization member has been deleted.')) #modified by HDX
+                self._action('group_member_delete')(
+                    context, {'id': id, 'user_id': user_id})
+                # modified by HDX
+                h.flash_notice(_('Organization member has been deleted.'))
                 self._redirect_to(controller='group', action='members', id=id)
             c.user_dict = self._action('user_show')(context, {'id': user_id})
             c.user_id = user_id
@@ -139,4 +141,5 @@ def member_delete(self, id):
             abort(401, _('Unauthorized to delete group %s') % '')
         except NotFound:
             abort(404, _('Group not found'))
-        self._redirect_to(controller='group', action='members', id=id) #modified by HDX
+        # modified by HDX
+        self._redirect_to(controller='group', action='members', id=id)


Test patch:
Repository: OCHA-DAP/hdx-ckan

Issue:
Update data on the Ebola map



Discussion:

Resolving patch:
diff --git a/ckanext-hdx_theme/ckanext/hdx_theme/version.py b/ckanext-hdx_theme/ckanext/hdx_theme/version.py
--- a/ckanext-hdx_theme/ckanext/hdx_theme/version.py
+++ b/ckanext-hdx_theme/ckanext/hdx_theme/version.py
@@ -1 +1 @@
-hdx_version = 'v0.5.13'
+hdx_version = 'v0.5.14'


Test patch:
Repository: OCHA-DAP/hdx-ckan

Issue:
Login page: change spacing on left panel 
The spacing in the left panel is odd.  Change to something like the below: 

![image](https://cloud.githubusercontent.com/assets/1654485/5451121/6d3fa152-850d-11e4-87dd-4485c2f767da.png)

Note, this will stay in backlog for now as we may want to revise this page to align with the Frog design.



Discussion:
@cjhendrix - should I also change the registration page? check screenshot
![screen shot 2015-01-08 at 16 23 13](https://cloud.githubusercontent.com/assets/3865844/5663699/c9274f90-9752-11e4-90bb-23d16d0eb1a0.png)

That one is not as confusing, but yes, why not make them consistent while we are at it.  In the case of registration, I'd just remove the dashed line completely.


Resolving patch:
diff --git a/ckanext-hdx_theme/ckanext/hdx_theme/version.py b/ckanext-hdx_theme/ckanext/hdx_theme/version.py
--- a/ckanext-hdx_theme/ckanext/hdx_theme/version.py
+++ b/ckanext-hdx_theme/ckanext/hdx_theme/version.py
@@ -1 +1 @@
-hdx_version = 'v0.5.13'
+hdx_version = 'v0.5.15'


Test patch:
Repository: OCHA-DAP/hdx-ckan

Issue:
ckanext-sitemap and ckanext-hdx_theme are missing top level __init__.py
we need this to be able to run ckan in production mode.



Discussion:

Resolving patch:
diff --git a/ckanext-hdx_theme/__init__.py b/ckanext-hdx_theme/__init__.py
new file mode 100644
diff --git a/ckanext-sitemap/__init__.py b/ckanext-sitemap/__init__.py
new file mode 100644


Test patch:
Repository: OCHA-DAP/hdx-ckan

Issue:
Problem with tracking summary numbers
The tracking key is the for resources is the "url". Since we switched to using the "perma_link" this has caused some issues since the data is stored in the tracking table with the perma_link but when showing a dataset or resource CKAN tries to do the mapping via the old URL. 



Discussion:

Resolving patch:
diff --git a/ckanext-hdx_package/ckanext/hdx_package/helpers/tracking_changes.py b/ckanext-hdx_package/ckanext/hdx_package/helpers/tracking_changes.py
new file mode 100644
--- /dev/null
+++ b/ckanext-hdx_package/ckanext/hdx_package/helpers/tracking_changes.py
@@ -0,0 +1,25 @@
+'''
+Created on Jan 19, 2015
+
+@author:alexandru-m-g
+'''
+
+import ckan.model as model
+
+def add_tracking_summary_to_resource_dict(resource_dict):
+    '''
+    This function supplements _add_tracking_summary_to_resource_dict()
+    from ckan/logic/action/get.py. It also checks to see if there is data
+    for the new permalink.
+    '''
+    url = resource_dict.get('perma_link', None)
+    if url:
+        tracking_summary = model.TrackingSummary.get_for_resource(url)
+        if tracking_summary:
+            if 'tracking_summary' in resource_dict \
+                    and resource_dict['tracking_summary']:
+                summary = resource_dict['tracking_summary']
+                summary['total'] = summary.get('total', 0) + tracking_summary.get('total', 0)
+                summary['recent'] = summary.get('recent', 0) + tracking_summary.get('recent', 0)
+            else:
+                resource_dict['tracking_summary'] = tracking_summary
diff --git a/ckanext-hdx_package/ckanext/hdx_package/plugin.py b/ckanext-hdx_package/ckanext/hdx_package/plugin.py
--- a/ckanext-hdx_package/ckanext/hdx_package/plugin.py
+++ b/ckanext-hdx_package/ckanext/hdx_package/plugin.py
@@ -19,6 +19,7 @@
 import ckanext.hdx_package.helpers.update as update
 import ckanext.hdx_package.actions.authorize as authorize
 import ckanext.hdx_package.helpers.helpers as hdx_helpers
+import ckanext.hdx_package.helpers.tracking_changes as tracking_changes
 
 
 def run_on_startup():
@@ -58,6 +59,7 @@ class HDXPackagePlugin(plugins.SingletonPlugin, tk.DefaultDatasetForm):
     plugins.implements(plugins.ITemplateHelpers)
     plugins.implements(plugins.IActions)
     plugins.implements(plugins.IMiddleware, inherit=True)
+    plugins.implements(plugins.IResourceController, inherit=True)
 
     def update_config(self, config):
         tk.add_template_directory(config, 'templates')
@@ -214,6 +216,14 @@ def get_actions(self):
             'package_create': hdx_actions.package_create
         }
 
+    def before_show(self, resource_dict):
+        '''
+            This is run before a resource is displayed.
+            We use it to show the correct tracking summary
+        '''
+        tracking_changes.add_tracking_summary_to_resource_dict(resource_dict)
+        return resource_dict
+
     def get_auth_functions(self):
         return {'package_create': authorize.package_create,
                 'package_update': authorize.package_update}


Test patch:
Repository: OCHA-DAP/hdx-ckan

Issue:
Prod bug: signup was not possible for different email addresses
Reported by user
It seems like there is a timeout when doing the email validation step, more precisely when checking the DNS lookup for the hostname. 
A quick fix might be to put check_mx=False  (instead of True).

Update data on the Ebola map



Discussion:

Resolving patch:
diff --git a/ckanext-hdx_search/ckanext/hdx_search/controllers/search_controller.py b/ckanext-hdx_search/ckanext/hdx_search/controllers/search_controller.py
--- a/ckanext-hdx_search/ckanext/hdx_search/controllers/search_controller.py
+++ b/ckanext-hdx_search/ckanext/hdx_search/controllers/search_controller.py
@@ -158,32 +158,12 @@ def isolate_features(context, facets, q, tab, skip=0, limit=25):
         return (feature_list[skip:skip + limit], len(features))
     return features
 
-def convert_country(q, country_names, countries):
-    print country_names
-    for c in country_names:
-        if c in q.lower():
-            q += ' '+countries[c]
-    print q
-    return q
-
-def build_country_dict(countries):
-    country_names = list()
-    country_dict = dict()
-    for c in countries:
-        country_names.append(c['display_name'].lower())
-        country_dict[c['display_name'].lower()] = c['name']
-    return (country_names, country_dict)
 
 class HDXSearchController(PackageController):
 
     def search(self):
         from ckan.lib.search import SearchError
-        context = {'model': model, 'session': model.Session,
-                   'user': c.user or c.author, 'for_view': True,
-                   'with_private': False}
 
-        data_dict = {'all_fields': True}
-        country_names, countries = build_country_dict(get_action('group_list')(context, data_dict))
         params_to_skip = ['_show_filters']
 
         package_type = self._guess_package_type()
@@ -200,7 +180,6 @@ def search(self):
 
         # unicode format (decoded from utf8)
         q = c.q = request.params.get('q', u'')
-        q = convert_country(q, country_names, countries)
         c.query_error = False
         try:
             page = int(request.params.get('page', 1))
diff --git a/ckanext-hdx_theme/ckanext/hdx_theme/version.py b/ckanext-hdx_theme/ckanext/hdx_theme/version.py
--- a/ckanext-hdx_theme/ckanext/hdx_theme/version.py
+++ b/ckanext-hdx_theme/ckanext/hdx_theme/version.py
@@ -1 +1 @@
-hdx_version = 'v0.6.3'
+hdx_version = 'v0.6.4'


Test patch:
Repository: huggingface/accelerate

Issue:
Can't send the values of int to device
My training data looks like:

src_image, target_image, src_camera, target_camera, src_camera_idx, target_camera_idx

Where src_camera_idx, target_camera_idx are integers

When I try to apply accelerate I get the following error:
TypeError: Can't send the values of type <class 'int'> to device cuda:0, only of nested list/tuple/dicts of tensors or objects having a `to` method.

We don't need to send the integers to the device. Perhaps instead of raising an error here, you can simply skip the items that cannot be moved to device? Or at least give me the option to skip them if I know my data has such objects.



Discussion:
@sgugger 
Oh indeed, I did not take that case into account. Not raising an error seems a better option.
Will send a fix later this evening or tomorrow morning!
Awesome, thanks

Resolving patch:
diff --git a/src/accelerate/utils.py b/src/accelerate/utils.py
--- a/src/accelerate/utils.py
+++ b/src/accelerate/utils.py
@@ -105,10 +105,7 @@ def send_to_device(tensor, device):
     elif isinstance(tensor, dict):
         return type(tensor)({k: send_to_device(v, device) for k, v in tensor.items()})
     elif not hasattr(tensor, "to"):
-        raise TypeError(
-            f"Can't send the values of type {type(tensor)} to device {device}, only of nested list/tuple/dicts "
-            "of tensors or objects having a `to` method."
-        )
+        return tensor
     return tensor.to(device)
 
 


Test patch:
Repository: huggingface/accelerate

Issue:
AcceleratedOptimizer `zero_grad` argument not supported: `set_to_none`
Currently the `AcceleratedOptimizer` class doesn't support the argument `set_to_none`, is this an intentional exclusion?


Discussion:
It's not intended, I actually wasn't aware PyTorch had added it in 1.7.0. Will just need to be careful to only accept it when the version of PyTorch is recent enough but support for it should be pretty straightforward otherwise!

Resolving patch:
diff --git a/src/accelerate/optimizer.py b/src/accelerate/optimizer.py
--- a/src/accelerate/optimizer.py
+++ b/src/accelerate/optimizer.py
@@ -14,6 +14,8 @@
 
 import torch
 
+from packaging import version
+
 from .state import AcceleratorState, DistributedType, is_tpu_available
 
 
@@ -86,8 +88,18 @@ def load_state_dict(self, state_dict):
     def state_dict(self):
         return self.optimizer.state_dict()
 
-    def zero_grad(self):
-        self.optimizer.zero_grad()
+    def zero_grad(self, set_to_none=None):
+        if version.parse(torch.__version__) < version.parse("1.7.0"):
+            if set_to_none is not None:
+                raise ValueError(
+                    "`set_to_none` for Optimizer.zero_grad` was introduced in PyTorch 1.7.0 and can't be used for "
+                    f"earlier versions (found version {torch.__version__})."
+                )
+            self.optimizer.zero_grad()
+        else:
+            if set_to_none is not None:
+                set_to_none = False
+            self.optimizer.zero_grad(set_to_none=set_to_none)
 
     def step(self):
         if self.state.distributed_type == DistributedType.TPU:


Test patch:
Repository: huggingface/accelerate

Issue:
Mismatch between `accelerate config` cli and  `default_config.yaml`
The generated `default_config.yaml` is mismatch with `accelerate config`.

Here are my cli outputs and `default_config.yaml`

cli outputs

```
In which compute environment are you running? ([0] This machine, [1] AWS (Amazon SageMaker)): 0
Which type of machine are you using? ([0] No distributed training, [1] multi-GPU, [2] TPU): 1
How many different machines will you use (use more than 1 for multi-node training)? [1]: 2
What is the rank of this machine (from 0 to the number of machines - 1 )? [0]: 1
What is the IP address of the machine that will host the main process? 10.29.150.50
What is the port you will use to communicate with the main process? 2333
How many processes in total will you use? [1]: 6
Do you wish to use FP16 (mixed precision)? [yes/NO]: yes

```

`default_config.yaml`
```
compute_environment: LOCAL_MACHINE
distributed_type: MULTI_GPU
fp16: true
machine_rank: 1
main_process_ip: 2333
main_process_port: null
main_training_function: main
num_machines: 2
num_processes: 6
```



Discussion:

Resolving patch:
diff --git a/src/accelerate/commands/config/cluster.py b/src/accelerate/commands/config/cluster.py
--- a/src/accelerate/commands/config/cluster.py
+++ b/src/accelerate/commands/config/cluster.py
@@ -46,7 +46,7 @@ def get_cluster_input():
             main_process_ip = _ask_field(
                 "What is the IP address of the machine that will host the main process? ",
             )
-            main_process_ip = _ask_field(
+            main_process_port = _ask_field(
                 "What is the port you will use to communicate with the main process? ",
                 lambda x: int(x),
             )


Test patch:
Repository: huggingface/accelerate

Issue:
Add warning not to reuse `Accelerator`
I was training multiple models, and there seems to be some GPU memory leak.
I think it's because `Accelerator` appends reference to optimizer, which in turn reference some GPU memory.
https://github.com/huggingface/accelerate/blob/df260fa71af9c23587f6c0a86df5ad17f69822d2/src/accelerate/accelerator.py#L164


When reusing `Accelerator` to train multiple models, the GPU memory used by old optimizer is not released.


Discussion:
Good point. I will add some sort of cleanup method that will release all references.

Resolving patch:
diff --git a/src/accelerate/accelerator.py b/src/accelerate/accelerator.py
--- a/src/accelerate/accelerator.py
+++ b/src/accelerate/accelerator.py
@@ -12,6 +12,7 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
+import gc
 from typing import List, Optional, Union
 
 import torch
@@ -338,6 +339,15 @@ def save(self, obj, f):
         """
         save(obj, f)
 
+    def free_memory(self):
+        """
+        Will release all references to the internal objects stored and call the garbage collector. You should call this
+        method between two trainings with different models/optimizers.
+        """
+        self._optimizers = []
+        gc.collect()
+        torch.cuda.empty_cache()
+
     def _get_named_parameters(self, *args):
         named_parameters = {}
         for obj in args:


Test patch:
Repository: huggingface/accelerate

Issue:
Accelerator backward with retain graph
I want to use `loss.backward(retain_graph=True)` instead of normal backward. How can I do this using accelerate?


Discussion:
Ah, this is currently not implemented, will make a PR to add support for this, good point!

Resolving patch:
diff --git a/src/accelerate/accelerator.py b/src/accelerate/accelerator.py
--- a/src/accelerate/accelerator.py
+++ b/src/accelerate/accelerator.py
@@ -387,16 +387,16 @@ def prepare_data_loader(self, data_loader):
     def prepare_optimizer(self, optimizer):
         return AcceleratedOptimizer(optimizer, device_placement=self.device_placement, scaler=self.scaler)
 
-    def backward(self, loss):
+    def backward(self, loss, **kwargs):
         """
         Use :obj:`accelerator.backward(loss)` in lieu of :obj:`loss.backward()`.
         """
         if self.distributed_type == DistributedType.DEEPSPEED:
-            self.deepspeed_engine.backward(loss)
+            self.deepspeed_engine.backward(loss, **kwargs)
         elif self.scaler is not None:
-            self.scaler.scale(loss).backward()
+            self.scaler.scale(loss).backward(**kwargs)
         else:
-            loss.backward()
+            loss.backward(**kwargs)
 
     def unscale_gradients(self, optimizer=None):
         """


Test patch:
Repository: huggingface/accelerate

Issue:
Unscaling Wrapped optimizer making the gradient is unscaled twice
Hi, thank you for the useful tool. 

https://github.com/huggingface/accelerate/blob/main/src/accelerate/accelerator.py#L416-L417

I'm referring to these lines of code where the wrapped optimizer is unscaled when gradient clipping. I think the inner optimizer  should be unscaled instead. Since the scaler will mark the optimizer state based on its id, giving wrapped optimizer and unscaling it would make the gradient get unscaled but the inner optimizer is not marked as unscaled. Then when calling `accelerator.backward()`, gradient get unscaled again.




Discussion:
That's right. Do you want to make a PR to fix this?
It is ok, you can fix it

Resolving patch:
diff --git a/src/accelerate/accelerator.py b/src/accelerate/accelerator.py
--- a/src/accelerate/accelerator.py
+++ b/src/accelerate/accelerator.py
@@ -413,8 +413,10 @@ def unscale_gradients(self, optimizer=None):
                 optimizer = self._optimizers
             elif not isinstance(optimizer, (tuple, list)):
                 optimizer = [optimizer]
-            for optimizer in optimizer:
-                self.scaler.unscale_(optimizer)
+            for opt in optimizer:
+                while isinstance(opt, AcceleratedOptimizer):
+                    opt = opt.optimizer
+                self.scaler.unscale_(opt)
 
     def clip_grad_norm_(self, parameters, max_norm, norm_type=2):
         """


Test patch:
Repository: huggingface/accelerate

Issue:
Error when loading optimizer state
Thanks for this awesome product!!

When I try to load optimizer state on TPUs, I get an error, since self.device_placement is never initialized in AcceleratedOptimizer..!
https://github.com/huggingface/accelerate/blob/e0a420f7cb32124cadeeae690b56e463f8fc598f/src/accelerate/optimizer.py#L83-L86
Error when loading optimizer state
Thanks for this awesome product!!

When I try to load optimizer state on TPUs, I get an error, since self.device_placement is never initialized in AcceleratedOptimizer..!
https://github.com/huggingface/accelerate/blob/e0a420f7cb32124cadeeae690b56e463f8fc598f/src/accelerate/optimizer.py#L83-L86


Discussion:

Resolving patch:
diff --git a/src/accelerate/optimizer.py b/src/accelerate/optimizer.py
--- a/src/accelerate/optimizer.py
+++ b/src/accelerate/optimizer.py
@@ -52,6 +52,7 @@ def __init__(self, optimizer, device_placement=True, scaler=None):
         self.optimizer = optimizer
         self.scaler = scaler
         self.state = AcceleratorState()
+        self.device_placement = device_placement
 
         # Handle device placement
         if device_placement:


Test patch: