Repository: hydroshare/hydroshare

Issue:
Sitemap.xml contains private resources
**Description of the bug**
A clear and concise description of what the bug is.

The sitemap.xml refers to the list of resources in a file [here ](https://www.hydroshare.org/sitemap-resources.xml ) by design this should only contain published, public and discoverable  resources but does also contain private resources e.g.[https://www.hydroshare.org/resource/e2fdf99cbb0c4275b32afd3c16ae6863/](https://www.hydroshare.org/resource/e2fdf99cbb0c4275b32afd3c16ae6863/). 

A fix need to be applied to remove all references to private resources.


Discussion:
I also see that our `robots.txt` lists an incorrect sitemap:
![image](https://github.com/hydroshare/hydroshare/assets/17934193/ecb2667b-5a51-4397-8ec5-18683d48cc23)
Looks like this has been the case since https://github.com/hydroshare/hydroshare/pull/1821

Seems we should be setting `ROBOTS_SITEMAP_URLS` in local_settings.py
https://github.com/hydroshare/hydroshare/pull/1821#issuecomment-277056182

From django-robots:
https://django-robots.readthedocs.io/en/latest/?highlight=ROBOTS_SITEMAP_URLS#sitemaps


Resolving patch:
diff --git a/hs_sitemap/sitemaps.py b/hs_sitemap/sitemaps.py
--- a/hs_sitemap/sitemaps.py
+++ b/hs_sitemap/sitemaps.py
@@ -1,13 +1,53 @@
 from django.contrib import sitemaps
 from django.urls import reverse
+from hs_core.models import BaseResource
+from hs_access_control.models import Community
+from django.contrib.auth.models import Group
+from django.db.models import Q
 
 
 class PagesSitemap(sitemaps.Sitemap):
     priority = 0.5
-    changefreq = 'daily'
+    changefreq = 'weekly'
 
     def items(self):
         return ['home', 'login', 'apps']
 
     def location(self, item):
         return reverse(item)
+
+
+class CommunitiesSitemap(sitemaps.Sitemap):
+    priority = 0.5
+    changefreq = 'weekly'
+
+    def items(self):
+        return Community.objects.filter(active=True)
+
+    def location(self, item):
+        return f'/community/{ item.id }'
+
+
+class GroupsSitemap(sitemaps.Sitemap):
+    priority = 0.5
+    changefreq = 'daily'
+
+    def items(self):
+        return Group.objects.filter(gaccess__active=True)
+
+    def location(self, item):
+        return f'/group/{ item.id }'
+
+
+class ResourcesSitemap(sitemaps.Sitemap):
+    priority = 0.5
+    changefreq = 'hourly'
+
+    def items(self):
+        return BaseResource.objects.filter(Q(raccess__public=True) | Q(raccess__discoverable=True))
+
+    def location(self, item):
+        return item.absolute_url
+
+    def lastmod(self, item):
+        return item.last_updated
diff --git a/hs_sitemap/urls.py b/hs_sitemap/urls.py
--- a/hs_sitemap/urls.py
+++ b/hs_sitemap/urls.py
@@ -1,13 +1,14 @@
 from django.conf.urls import url
 
 from django.contrib.sitemaps import views
-from mezzanine.core.sitemaps import DisplayableSitemap
 
-from .sitemaps import PagesSitemap
+from .sitemaps import PagesSitemap, ResourcesSitemap, CommunitiesSitemap, GroupsSitemap
 
 sitemaps = {
-    "resources": DisplayableSitemap,
-    "pages": PagesSitemap
+    "resources": ResourcesSitemap,
+    "pages": PagesSitemap,
+    "communities": CommunitiesSitemap,
+    "groups": GroupsSitemap
 }
 sitemap_view = 'django.contrib.sitemaps.views.sitemap'
 
diff --git a/hydroshare/settings.py b/hydroshare/settings.py
--- a/hydroshare/settings.py
+++ b/hydroshare/settings.py
@@ -178,11 +178,6 @@ def _(s): return s # noqa
 # Alternative tmp folder
 FILE_UPLOAD_TEMP_DIR = "/hs_tmp"
 
-# Sitemap for robots
-ROBOTS_SITEMAP_URLS = [
-    "http://localhost:8000/sitemap/",
-]
-
 #############
 # DATABASES #
 #############


Test patch: