internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: OpenEnergyPlatform/oeplatform

Issue:
Tutorials app: activate safe_mode
Html is not escaped as expected when creating a new tutorial using the markdown editor -> save tutorial.


Discussion:

Resolving patch:
diff --git a/tutorials/views.py b/tutorials/views.py
--- a/tutorials/views.py
+++ b/tutorials/views.py
@@ -214,7 +214,11 @@ def formattedMarkdown(markdown):
     :return:
     """
 
-    markdowner = Markdown(safe_mode="escape")
+    # escapes html but also escapes html code blocks lke "exampel code:
+    #                                                    (1 tab)  code"
+    # checkbox also not rendered as expected "- [ ]"
+    markdowner = Markdown(safe_mode=True)
+    markdowner.html_removed_text = ""
 
     return markdowner.convert(markdown)
 


Test patch:
Repository: OpenEnergyPlatform/oeplatform

Issue:
Page dataedit should no longer exist
Dataedit contains a link to the schemas and to the API documentation. The API documentation can be linked to on the frontpage where the link to dataedit sits now. The link to the schemas should replace all existing links to dataedit.


Discussion:

Resolving patch:
diff --git a/dataedit/urls.py b/dataedit/urls.py
--- a/dataedit/urls.py
+++ b/dataedit/urls.py
@@ -1,12 +1,13 @@
 from django.conf.urls import url
 
 from dataedit import views
+from django.views.generic import RedirectView
 
 pgsql_qualifier = r"[\w\d_]+"
 
 urlpatterns = [
     url(r"^schemas$", views.listschemas, name="index"),
-    url(r"^$", views.overview, name="index"),
+    url(r"^$", RedirectView.as_view(url='/dataedit/schemas')),
     # url(r'^admin/$', views.admin, name='index'),
     url(r"^admin/columns/", views.admin_columns, name="input"),
     url(r"^admin/constraints/", views.admin_constraints, name="input"),
diff --git a/dataedit/views.py b/dataedit/views.py
--- a/dataedit/views.py
+++ b/dataedit/views.py
@@ -296,10 +296,6 @@ def listschemas(request):
     )
 
 
-def overview(request):
-    return render(request, "dataedit/dataedit_choices.html", {})
-
-
 def read_label(table, comment):
     """
     Extracts the readable name from @comment and appends the real name in parens.
@@ -1498,4 +1494,4 @@ def get(self, request, schema='model_draft', table=None):
             "table": table
         }
 
-        return render(request, "dataedit/wizard.html", context=context)
\ No newline at end of file
+        return render(request, "dataedit/wizard.html", context=context)


Test patch:
Repository: OpenEnergyPlatform/oeplatform

Issue:
toep: editing metadata pressing 'submit' throws error
when editing metadata (worked before), clicking the submit button throws a pop-up message which states 'forbidden'.

Suggested improvement: the submit button may also be placed at the top. Now there is only one at the button and the form is really long. 


Discussion:
@wingechr: I checked the permissions, and they do not seem to be an issue. Maybe the way the queries are sent is an issue. Do they pass the right credentials to the API?
PS: If `oeko_testtable` was used in this case.

Resolving patch:
diff --git a/dataedit/views.py b/dataedit/views.py
--- a/dataedit/views.py
+++ b/dataedit/views.py
@@ -941,10 +941,12 @@ def get(self, request, schema, table):
         display_items = api_changes.get("display_items")
 
         is_admin = False
-        can_write = False
+        can_add = False # can upload data
+        table_obj = Table.load(schema, table)
         if request.user and not request.user.is_anonymous:
             is_admin = request.user.has_admin_permissions(schema, table)
-            can_write = request.user.has_write_permissions(schema, table)
+            level = request.user.get_table_permission_level(table_obj)
+            can_add = level >= login_models.WRITE_PERM
 
         table_views = DBView.objects.filter(table=table).filter(schema=schema)
 
@@ -980,7 +982,7 @@ def get(self, request, schema, table):
             "filter": current_view.filter.all(),
             "current_view": current_view,
             "is_admin": is_admin,
-            "can_add": can_write,
+            "can_add": can_add,
             "host": request.get_host(),
         }
 
@@ -1375,7 +1377,8 @@ def get(self, request, schema='model_draft', table=None):
                 "nRows": n_rows
             }),
             "schema": schema,
-            "table": table
+            "table": table,
+            "can_add": can_add
         }
 
         return render(request, "dataedit/wizard.html", context=context)
@@ -1388,6 +1391,12 @@ def get(self, request, schema, table):
 
         columns = get_column_description(schema, table)
 
+        can_add = False
+        table_obj = Table.load(schema, table)
+        if not request.user.is_anonymous:
+            level = request.user.get_table_permission_level(table_obj)
+            can_add = level >= login_models.WRITE_PERM
+
         context_dict = {
             "config": json.dumps({
                 "schema": schema,
@@ -1395,7 +1404,8 @@ def get(self, request, schema, table):
                 "columns": columns,
                 "url_api_meta": reverse('api_table_meta', kwargs={"schema": schema, "table": table}),
                 "url_view_table": reverse('view', kwargs={"schema": schema, "table": table}),
-            })
+            }),
+            "can_add": can_add
         }
 
         return render(


Test patch:
Repository: OpenEnergyPlatform/oeplatform

Issue:
Database: Open data only.
Where do we need / want to remind the users?
I added this rule to the [DatabaseRules](http://wiki.openmod-initiative.org/wiki/DatabaseRules).
Do we want to add a reminder to the upload interface?



Discussion:
good idea.
Perhaps use a checkbox:
[] I confirm that I only upload Open Data...

good examples for code that can be adapted:

- http://www.eclipse.org/legal/CoO.php
- http://www.eclipse.org/legal/ECA.php
- http://www.eclipse.org/legal/DCO.php

Before updating data to the oedb via the OEP there must be a checkbox to confirm the **OEP Certificate of Origin**

>  Copyright (C) 2004, 2006 The Linux Foundation and its contributors.
> 660 York Street, Suite 102,
> San Francisco, CA 94110 USA
> 
> Everyone is permitted to copy and distribute verbatim copies of this license document, **but changing it is not allowed.** 

Damn, I just changed it.
RequirementSpecificationID=12
I changed the copyright in Jupyter-notebook but, can you please confirm the change?
I attached the screen-shot of Jupyter-notebook. 

![Screenshot (15)](https://user-images.githubusercontent.com/52157695/60009054-76cd0b00-9675-11e9-93dd-1f5e60de9c6d.png)
![Screenshot (14)](https://user-images.githubusercontent.com/52157695/60009055-76cd0b00-9675-11e9-98fa-2131a0f6d722.png)


@Dhruvin14 - I am not sure whether changing the copyright in the OEP jupyter notebook tutorials is addressing this issue. As I understood it, this copyright is linked to the institution of the tutorials' author 
I am also not sure as to why this change is necessary, or related to this issue. Could you elaborate @Dhruvin14 ?
@Bachibouzouk 
before I understood that I need to add copyrights and licence in each tutorial where user create the database from tutorials. 
But, as we discussed, It is with uploading the metadata on the OE interface, and it seems to finish by martin. 
I will confirm the changes and let you know. 
This issue is confusing :confused: 
Is it already solved? Then,please close. Else, comment on what is missing @christian-rli or @Bachibouzouk. Thanks.

Resolving patch:
diff --git a/base/urls.py b/base/urls.py
--- a/base/urls.py
+++ b/base/urls.py
@@ -9,7 +9,6 @@
     url(r"^faq/$", views.redir, {"target": "faq"}, name="index"),
     url(r"^discussion/$", views.redir, {"target": "discussion"}, name="index"),
     url(r"^contact/$", views.ContactView.as_view(), name="index"),
-    url(r"^legal/impressum/$", views.redir, {"target": "impressum"}, name="index"),
-    url(r"^legal/datasec/$", views.redir, {"target": "datasecurity"}, name="index"),
+    url(r"^legal/privacy_policy/$", views.redir, {"target": "privacy_policy"}, name="index"),
     url(r"^legal/tou/$", views.redir, {"target": "terms_of_use"}, name="index"),
 ]


Test patch:
Repository: OpenEnergyPlatform/oeplatform

Issue:
Tutorial form: Markdown for code does not work
See example tutorial
https://openenergy-platform.org/tutorials/12/

This would be important for our beginners tutorial to work:
https://openenergy-platform.org/tutorials/11/
 

> See also #405 






Discussion:
I noticed this already, I will try to fix this before the upcoming release next week.  

Resolving patch:
diff --git a/tutorials/views.py b/tutorials/views.py
--- a/tutorials/views.py
+++ b/tutorials/views.py
@@ -147,7 +147,8 @@ def formattedMarkdown(markdown):
     # escapes html but also escapes html code blocks lke "exampel code:
     #                                                    (1 tab)  code"
     # checkbox also not rendered as expected "- [ ]"
-    markdowner = Markdown(safe_mode=True)
+    # TODO: Add syntax highliting, add css files -> https://github.com/trentm/python-markdown2/wiki/fenced-code-blocks 
+    markdowner = Markdown( extras=["break-on-newline", "fenced-code-blocks"], safe_mode=True)
     markdowner.html_removed_text = ""
 
     return markdowner.convert(markdown)
@@ -210,6 +211,7 @@ def form_valid(self, form):
 
         # Convert markdown to HTML and save to db
         _html = formattedMarkdown(tutorial.markdown)
+        print(_html)
         addHtml = Tutorial.objects.get(pk=tutorial.id)
         addHtml.html = _html
         addHtml.save()


Test patch:
Repository: OpenEnergyPlatform/oeplatform

Issue:
Schema reference needs an updated description
As long as we still have schemas, we may add "helper" tables in the "reference" schema. We need to update the description accordingly.

Suggestion:
Contains, sources, literature and helper tables which can assist with your work. 


Discussion:
If possible I would also rename the display name of the schema to
'reference and help'   (and keep the url)
Contains sources, literature and auxiliary/helper tables that can help you in your work.
So. let's implement for now: :
Contains sources, literature and auxiliary/helper tables that can help you with your work.

Resolving patch:
diff --git a/dataedit/views.py b/dataedit/views.py
--- a/dataedit/views.py
+++ b/dataedit/views.py
@@ -262,7 +262,7 @@ def listschemas(request):
         "society": "Demographic data such as population statistics and projections, fertility, mortality etc.",
         "model_draft": "Unfinished data of any kind. Note: there is no version control and data is still volatile.",
         "scenario": "Scenario data in the broadest sense. Includes input and output data from models that project scenarios into the future. Example inputs: assumptions made about future developments of key parameters such as energy prices and GDP. Example outputs: projected electricity transmission, projected greenhouse gas emissions. Note that inputs to one model could be an output of another model and the other way around.",
-        "reference": "sources, literature",
+        "reference": "Contains sources, literature and auxiliary/helper tables that can help you with your work.",
         "emission": "Data on emissions. Examples: total greenhouse gas emissions, CO2-emissions, energy-related CO2-emissions, methane emissions, air pollutants etc.",
         "openstreetmap": "OpenStreetMap is a open project that collects and structures freely usable geodata and keeps them in a database for use by anyone. This data is available under a free license, the Open Database License.",
         "policy": "Data on policies and measures. This could, for example, include a list of renewable energy policies per European Member State. It could also be a list of climate related policies and measures in a specific country."


Test patch:
Repository: OpenEnergyPlatform/oeplatform

Issue:
Meta-schemas are not created properly
It seems that a fresh installation of the most recent version does not create the meta-schemas anymore.


Discussion:

Resolving patch:
diff --git a/oedb_datamodels/versions/5c098aa81e2c_create_schemas.py b/oedb_datamodels/versions/5c098aa81e2c_create_schemas.py
--- a/oedb_datamodels/versions/5c098aa81e2c_create_schemas.py
+++ b/oedb_datamodels/versions/5c098aa81e2c_create_schemas.py
@@ -30,12 +30,15 @@
     "reference",
 ]
 
-
 def upgrade():
     for s in schemas:
         op.execute("CREATE SCHEMA " + s)
+    for s in schemas:
+        op.execute("CREATE SCHEMA _" + s)
 
 
 def downgrade():
     for s in schemas:
-        op.execute("DROP SCHEMA " + s)
+        op.execute("DROP SCHEMA _" + s + " CASCADE")
+    for s in schemas:
+        op.execute("DROP SCHEMA " + s + " CASCADE")


Test patch:
Repository: OpenEnergyPlatform/oeplatform

Issue:
adding tags with umlauts end up with truncated keywords
i.e. adding `Öko-Institute` adds keyword `ko_institute`


Discussion:
also noticed this: i think in the first place it is related to the [regex when normalizing the tag name](https://github.com/OpenEnergyPlatform/oeplatform/blob/dd22a23dc061eeea8d280b82274d04854bb341f3/dataedit/structures.py#L54), and also it  might be that I mixed something up and now the new keyword is created using the normalized tag name as keyword name.
yea, the keyword are the normalized names, which they should not
I will fix this soon. 

Resolving patch:
diff --git a/dataedit/views.py b/dataedit/views.py
--- a/dataedit/views.py
+++ b/dataedit/views.py
@@ -1326,6 +1326,27 @@ def get_tag_name_normalized_by_id(session, tag_id):
     else:
         return None
 
+def get_tag_name_by_id(session, tag_id):
+    """
+    Query the Tag table in schmea public to get the tags.name.
+    Tags are queried by tag id.
+
+    Args:
+        session (sqilachemy): sqlalachemy session
+        tag_id (int): The Tag ID
+
+    Returns:
+        None: If tag id does not exists.
+        Str: Tag name
+    """
+
+    tag = session.query(Tag).filter(Tag.id==tag_id).first()
+    session.commit()
+    if tag is not None:
+        return tag.name
+    else:
+        return None
+
 
 def add_existing_keyword_tag_to_table_tags(session, schema, table, keyword_tag_id):
     """
@@ -1391,7 +1412,7 @@ def process_oem_keywords(session, schema, table, tag_ids, removed_table_tag_ids,
     updated_keywords = []
 
     for id in tag_ids:
-        kw = get_tag_name_normalized_by_id(session, id)
+        kw = get_tag_name_by_id(session, id)
         if kw is not None:
            updated_oep_tags.append(kw)
 


Test patch:
Repository: OpenEnergyPlatform/oeplatform

Issue:
Permissions: Renaming a permission group is not possible
Existing groups cannot be renamed.


Discussion:

Resolving patch:
diff --git a/login/views.py b/login/views.py
--- a/login/views.py
+++ b/login/views.py
@@ -89,6 +89,7 @@ def post(self, request, group_id=None):
         form = GroupForm(request.POST, instance=group)
         if form.is_valid():
             if group_id:
+                group = form.save()
                 membership = get_object_or_404(
                     GroupMembership, group=group, user=request.user
                 )


Test patch:
Repository: OpenEnergyPlatform/oeplatform

Issue:
Lack of standalone metadata "generator"


As a   user, I want to be able to create metadata records quickly (with a metadata   generator).





Discussion:
RequirementSpecificationID=38
A good way of doing this is to build upon the form already supplied for editing metadata. Default values and fields can be generated when loading the page of a data set. Those values should include all the information that is included in the dataset itself - i.e. the name of the dataset and its resources, including data type. 
A metadata generator has been implemented for the csv upload wizard. Unfortunately it cannot really be used standalone. Changed the title of this issue accordingly. 

Resolving patch:
diff --git a/dataedit/urls.py b/dataedit/urls.py
--- a/dataedit/urls.py
+++ b/dataedit/urls.py
@@ -99,4 +99,9 @@
         views.WizardView.as_view(),
         name="wizard_create",
     ),
+    url(
+        r"^standalone_meta_edit/$",
+        views.StandaloneMetaEditView.as_view(),
+        name="standalone_meta_edit",
+    )
 ]
diff --git a/dataedit/views.py b/dataedit/views.py
--- a/dataedit/views.py
+++ b/dataedit/views.py
@@ -1725,6 +1725,8 @@ def get(self, request, schema='model_draft', table=None):
 
         return render(request, "dataedit/wizard.html", context=context)
 
+def get_cancle_state(request):
+            return request.META.get('HTTP_REFERER')
 
 class MetaEditView(LoginRequiredMixin, View):
     """Metadata editor (cliet side json forms)."""
@@ -1748,6 +1750,8 @@ def get(self, request, schema, table):
                 "url_table_id": url_table_id,
                 "url_api_meta": reverse('api_table_meta', kwargs={"schema": schema, "table": table}),
                 "url_view_table": reverse('view', kwargs={"schema": schema, "table": table}),
+                "cancle_url": get_cancle_state(self.request),
+                "standalone": False
             }),
             "can_add": can_add
         }
@@ -1757,3 +1761,19 @@ def get(self, request, schema, table):
             "dataedit/meta_edit.html",
             context=context_dict,
         )
+
+
+class StandaloneMetaEditView(LoginRequiredMixin, View):
+    def get(self, request):
+        
+        context_dict = {
+            "config": json.dumps({
+                "cancle_url": get_cancle_state(self.request),
+                "standalone": True
+            })
+        }
+        return render(
+            request,
+            "dataedit/meta_edit.html",
+            context=context_dict,
+        )


Test patch:
Repository: OpenEnergyPlatform/oeplatform

Issue:
Update to bootstrap 5 
It would be nice to update from bootstrap 4 to 5. 

We would have to upgrade the PyPi package `django-bootstrap4` to https://pypi.org/project/django-bootstrap5/
and also review all bootstrap related code and files: https://github.com/OpenEnergyPlatform/oeplatform/search?q=bootstrap


Discussion:

Resolving patch:
diff --git a/dataedit/urls.py b/dataedit/urls.py
--- a/dataedit/urls.py
+++ b/dataedit/urls.py
@@ -47,12 +47,6 @@
         views.MetaEditView.as_view(),
         name="meta_edit",
     ),
-    url(
-        r"^view/(?P<schema>{qual})/(?P<table>{qual})/view$".format(
-            qual=pgsql_qualifier
-        ),
-        views.view_edit,
-    ),
     url(
         r"^view/(?P<schema>{qual})/(?P<table>{qual})/view/save$".format(
             qual=pgsql_qualifier
diff --git a/dataedit/views.py b/dataedit/views.py
--- a/dataedit/views.py
+++ b/dataedit/views.py
@@ -787,31 +787,6 @@ def add_tag(name, color):
     session.commit()
 
 
-def view_edit(request, schema, table):
-    post_id = request.GET.get("id")
-    if post_id:
-        view = DBView.objects.get(id=post_id)
-        context = {
-            "type": view.type,
-            "view": view,
-            "schema": schema,
-            "table": table,
-            "filter": view.filter.all(),
-        }
-        if view.options is not None:
-            context.update(view.options)
-        return render(
-            request, template_name="dataedit/view_editor.html", context=context
-        )
-    else:
-        type = request.GET.get("type")
-        return render(
-            request,
-            template_name="dataedit/view_editor.html",
-            context={"type": type, "new": True, "schema": schema, "table": table},
-        )
-
-
 def view_save(request, schema, table):
     post_name = request.POST.get("name")
     post_type = request.POST.get("type")
diff --git a/oeplatform/settings.py b/oeplatform/settings.py
--- a/oeplatform/settings.py
+++ b/oeplatform/settings.py
@@ -36,7 +36,7 @@
     "django.contrib.messages",
     "django.contrib.staticfiles",
     "django.contrib.sessions.backends.signed_cookies",
-    "bootstrap4",
+    "django_bootstrap5",
     "rest_framework",
     "rest_framework.authtoken",
     "modelview",
@@ -54,7 +54,6 @@
     "django.contrib.postgres",
     "fontawesome_5",
     "tutorials",
-    "jquery",
     "django_better_admin_arrayfield",
     "oeo_viewer",
 )


Test patch:
Repository: OpenEnergyPlatform/oeplatform

Issue:
Multiple definitions of primary key error when defining PK on table column
See https://github.com/OpenEnergyPlatform/oedialect/issues/43 for reference

I will add a small PR to address this.


Discussion:

Resolving patch:
diff --git a/api/actions.py b/api/actions.py
--- a/api/actions.py
+++ b/api/actions.py
@@ -866,6 +866,8 @@ def table_create(schema, table, column_definitions, constraints_definitions):
                 ccolumns = [constraint["constraint_parameter"]]
 
             if primary_key_col_names:
+                if ccolumns in primary_key_col_names:
+                    continue
                 raise APIError("Multiple definitions of primary key")
             primary_key_col_names = ccolumns
 


Test patch:
Repository: OpenEnergyPlatform/oeplatform

Issue:
Improve APIError messages
Currently, error messages are partially unspecific when creating tables via the API, e.g. for table and column header names

- [x] improve err msg for wrong column headers [e8588e8](https://github.com/OpenEnergyPlatform/oeplatform/commit/e8588e8c89593c36f8a296f4c52d7b1443780edd)
- Same error messages implemented for all three checks to give user the full picture of possible errors, independent of other error types, to avoid piecewise error correction by the user.
- [x] improve err msg for table names 


Discussion:

Resolving patch:
diff --git a/api/actions.py b/api/actions.py
--- a/api/actions.py
+++ b/api/actions.py
@@ -810,10 +810,10 @@ def column_add(schema, table, column, description):
 def assert_valid_identifier_name(identifier):
     if not IDENTIFIER_PATTERN.match(identifier):
         raise APIError(
-            "Invalid name. "
-            "Names must consist of lowercase alpha-numeric words or underscores "
+            f"Unsupported table name: {identifier}\n"
+            "Table names must consist of lowercase alpha-numeric words or underscores "
             "and start with a letter "
-            "and must be have a maximumlength of %s" % MAX_IDENTIFIER_LENGTH
+            f"and must not exceed {MAX_IDENTIFIER_LENGTH} characters (current table name length: {len(identifier)})."
         )
 
 
diff --git a/api/views.py b/api/views.py
--- a/api/views.py
+++ b/api/views.py
@@ -52,6 +52,8 @@
 
 logger = logging.getLogger("oeplatform")
 
+MAX_COL_NAME_LENGTH = 50
+
 WHERE_EXPRESSION = re.compile(
     r"^(?P<first>[\w\d_\.]+)\s*(?P<operator>"
     + r"|".join(parser.sql_operators)
@@ -453,10 +455,20 @@ def put(self, request, schema, table):
 
     def validate_column_names(self, column_definitions):
         """Raise APIError if any column name is invalid"""
+
         for c in column_definitions:
             colname = c["name"]
+
+            err_msg = f"Unsupported column name: '{colname}'\nColumn name must consist of lowercase alpha-numeric " \
+                      f"words or underscores and start with a letter. It must not start with an underscore or exceed " \
+                      f"{MAX_COL_NAME_LENGTH} characters (current column name length: {len(colname)})." \
+
             if not colname.isidentifier():
-                raise APIError("Invalid column name: %s" % colname)
+                raise APIError(f"{err_msg}")
+            if re.search(r"[A-Z]", colname) or re.match(r"_", colname):
+                raise APIError(f"Column names must not contain capital letters or start with an underscore! {err_msg}")
+            if len(colname) > MAX_COL_NAME_LENGTH:
+                raise APIError(f"Column name is too long! {err_msg}")
 
     @load_cursor()
     def __create_table(


Test patch: