Repository: pydicom/pydicom

Issue:
Codify not generating content sequences correctly
**Describe the bug**
I am trying to generate a radiation dose structure report. I ran Codify on an existing RDSR to generate a template. The sequence content is reproduced but does not seem to be attached  to the base dataset. When I run the generated python file the dicom file it saves has no sequence content information.

**Expected behavior**
I expect the dicom file generated by the python code from Codify to be similar to the original file.

**Steps To Reproduce**
$ python codify X-RayRadiationDoseReport001_ESR.dcm rdsr.py
$ python rsdr.py

I am not able to attached the above files but can supply them.

**Your environment**
module       | version
------       | -------
platform     | Linux-5.18.7-200.fc36.x86_64-x86_64-with-glibc2.35
Python       | 3.10.5 (main, Jun  9 2022, 00:00:00) [GCC 12.1.1 20220507 (Red Hat 12.1.1-1)]
pydicom      | 2.3.0
gdcm         | _module not found_
jpeg_ls      | _module not found_
numpy        | 1.22.4
PIL          | 9.2.0
pylibjpeg    | _module not found_
openjpeg     | _module not found_
libjpeg      | _module not found_

Regards
Alan


Discussion:
Hi Alan,

I'm happy to look into this, but may need a file to work with.  But perhaps we can first have a look at the generated code, which normally immediately attaches any sequence it creates.  For example:

```
$ pydicom codify pydicom::rtplan.dcm
```
Gives lines like

```python
# Beam Sequence
beam_sequence = Sequence()
ds.BeamSequence = beam_sequence
```

and then proceeds to create a new Dataset, set its data elements, and eventually `append` it to the sequence:

```python
# Beam Sequence: Beam 1
beam1 = Dataset()
beam1.Manufacturer = 'Linac co.'
beam1.InstitutionName = 'Here'
.
.
.
beam_sequence.append(beam1)
```
Can you check your generated code for those kinds of patterns and let us know what you see?


Hi Darcy

Thanks for the guidance. Stepping through the code I've picked up two problems. The first has to do with nested content. Eg. if the DICOM file has:
![NestedContent](https://user-images.githubusercontent.com/34305581/185054806-ed2559c6-2e0f-46e5-a9de-085ada6e7753.png)

The second Content1 overwrites the first. See lines 121-191 in the attached file.

[rdsr.txt](https://github.com/pydicom/pydicom/files/9356766/rdsr.txt) Rename this file to rsdr.py.

The second problem is more puzzling. In lines 121-191 we have:

```
content_sequence = Sequence()
ds.ContentSequence = content_sequence

# Content Sequence: Content 1
content1 = Dataset()
content1.add_new((0x0040, 0x0000), 'UL', 512)
content1.RelationshipType = 'HAS CONCEPT MOD'
content1.ValueType = 'CODE'
.
.
.
content_sequence.append(content1)
```
After stepping though this ds.ContentSequence is still empty even though there are definitely items in content1 and the items were added to content_sequence. It is not related to the nested content as content2 (lines 193 - 223 in rdsr.py) which has no nested content shows the same behaviour.

If I shift the line "ds.ContentSequence = content_sequence" to just before the save statement at the end of the file the script crashes while saving with:
RecursionError: maximum recursion depth exceeded while calling a Python object
This suggests two things:
1) The previous code is generating a nested loop somewhere.
2) The line "ds.ContentSequence = content_sequence" appears to be creating a copy of content_sequence. My understanding of python is that an assigning objects should merely update pointers, but this does not appear to be the case.

I would suggest generating content names that reflect the content level if possible, eg. content1-1.

Regards
Alan
Alan, if you are unable to share your RDSR for testing, do you want to check the behaviour is the same with one of the OpenREM test RDSR files and use that for Darcy or anyone else to work on? One of the more straight forward ones, maybe [this Siemens Flash one](https://bitbucket.org/openrem/openrem/src/develop/openrem/remapp/tests/test_files/CT-RDSR-Siemens_Flash-TAP-SS.dcm)?
Hi Ed
I ran Codify on  CT-RDSR-Siemens_Flash-TAP-SS.dcm as you suggested. This gave the attached file:
[CT-RDSR-Siemens_Flash-TAP-SS.py.txt](https://github.com/pydicom/pydicom/files/9359044/CT-RDSR-Siemens_Flash-TAP-SS.py.txt) rename to .py
Running this file produces a dicom file 10 times smaller than the original file (attached)
[CT-RDSR-Siemens_Flash-TAP-SS_from_codify.dcm.txt](https://github.com/pydicom/pydicom/files/9359049/CT-RDSR-Siemens_Flash-TAP-SS_from_codify.dcm.txt) Rename to .dcm
Looking at this in a dicom viewer the sequence content is gone.
Hope this helps
Regards
Alan



> The second Content1 overwrites the first.
> ...
> I would suggest generating content names that reflect the content level if possible, eg. content1-1.

Right, it makes sense - the code didn't consider that a nested item could have the same name.  I can update that.


> RecursionError: maximum recursion depth exceeded while calling a Python object

Not too surprising - I think I hit recursion problems when I first wrote codify.

Thanks for the detailed investigation and example files (thanks also @edmcdonagh  for the suggestion).  I can set up some failing tests with those and then see about fixing them.  I've got some time in the next couple of days, hopefully it isn't too difficult and I can get a PR soon.

Patch:
diff --git a/pydicom/cli/main.py b/pydicom/cli/main.py
--- a/pydicom/cli/main.py
+++ b/pydicom/cli/main.py
@@ -10,10 +10,11 @@
 import argparse
 import pkg_resources
 import re
+import sys
 from typing import Tuple, cast, List, Any, Dict, Optional, Callable
 
 from pydicom import dcmread
-from pydicom.data.data_manager import get_testdata_file
+from pydicom.data.data_manager import get_charset_files, get_testdata_file
 from pydicom.dataset import Dataset
 
 
@@ -132,6 +133,14 @@ def filespec_parser(filespec: str) -> List[Tuple[Dataset, Any]]:
     except NotImplementedError:  # will get this if absolute path passed
         pydicom_filename = ""
 
+    # Check if filename is in charset files
+    if not pydicom_filename:
+        try:
+            char_filenames = get_charset_files(filename)
+            pydicom_filename = char_filenames[0]
+        except NotImplementedError:  # will get this if absolute path passed
+            pass
+
     if prefix == "pydicom":
         filename = pydicom_filename
 
@@ -201,8 +210,11 @@ def main(args: Optional[List[str]] = None) -> None:
     """
     global subparsers
 
+    py_version = sys.version.split()[0]
+
     parser = argparse.ArgumentParser(
-        prog="pydicom", description="pydicom command line utilities"
+        prog="pydicom",
+        description=f"pydicom command line utilities (Python {py_version})"
     )
     subparsers = parser.add_subparsers(help="subcommand help")
 
diff --git a/pydicom/util/codify.py b/pydicom/util/codify.py
--- a/pydicom/util/codify.py
+++ b/pydicom/util/codify.py
@@ -18,7 +18,8 @@
 import os.path
 import re
 import sys
-from typing import Optional, List, Callable
+from typing import Optional, List, Callable, cast
+from collections import deque
 
 import pydicom
 from pydicom.datadict import dictionary_keyword
@@ -80,7 +81,8 @@ def code_dataelem(
     dataelem: DataElement,
     dataset_name: str = "ds",
     exclude_size: Optional[int] = None,
-    include_private: bool = False
+    include_private: bool = False,
+    var_names: Optional[deque] = None
 ) -> str:
     """Code lines for a single DICOM data element
 
@@ -96,7 +98,8 @@ def code_dataelem(
         will only have a commented string for a value,
         causing a syntax error when the code is run,
         and thus prompting the user to remove or fix that line.
-
+    var_names: Union[deque, None]
+        Used internally to ensure unique variable names in nested sequences.
     Returns
     -------
     str
@@ -106,7 +109,8 @@ def code_dataelem(
 
     if dataelem.VR == VR.SQ:
         return code_sequence(
-            dataelem, dataset_name, exclude_size, include_private
+            dataelem, dataset_name, exclude_size, include_private,
+            var_names=var_names
         )
 
     # If in DICOM dictionary, set using the keyword
@@ -143,6 +147,7 @@ def code_sequence(
     exclude_size: Optional[int] = None,
     include_private: bool = False,
     name_filter: Callable[[str], str] = default_name_filter,
+    var_names: Optional[deque] = None,
 ) -> str:
     """Code lines for recreating a Sequence data element
 
@@ -162,6 +167,8 @@ def code_sequence(
     name_filter: Callable[[str], str]
         A callable taking a sequence name or sequence item name, and returning
         a shorter name for easier code reading
+    var_names: Union[deque, None]
+        Used internally to ensure unique variable names in nested sequences.
 
     Returns
     -------
@@ -169,6 +176,15 @@ def code_sequence(
         A string containing code lines to recreate a DICOM sequence
     """
 
+    # Normally var_names is given from code_dataset, but for some tests need
+    #   to initialize it
+    if var_names is None:
+        var_names = deque()
+
+    def unique_name(name: str) -> str:
+        name_count = cast(deque, var_names).count(name) - 1
+        return name if name_count == 0 else name + f"_{name_count}"
+
     lines = []
     seq = dataelem.value
     seq_name = dataelem.name
@@ -183,8 +199,11 @@ def code_sequence(
     lines.append("# " + seq_name)
 
     # Code line to create a new Sequence object
-    if name_filter:
-        seq_var = name_filter(seq_keyword)
+    seq_var = name_filter(seq_keyword)
+    var_names.append(seq_var)
+    orig_seq_var = seq_var
+    seq_var = unique_name(seq_var)
+
     lines.append(seq_var + " = Sequence()")
 
     # Code line to add the sequence to its parent
@@ -208,14 +227,29 @@ def code_sequence(
         lines.append("# " + seq_name + ": " + seq_item_name + " " + index_str)
 
         # Determine the variable name to use for the sequence item (dataset)
-        ds_name = seq_var.replace("_sequence", "") + index_str
+        ds_name = orig_seq_var.replace("_sequence", "") + index_str
 
-        # Code the sequence item
-        code_item = code_dataset(ds, ds_name, exclude_size, include_private)
-        lines.append(code_item)
+        # Append "_#" if name already in use (in parent sequences)
+        var_names.append(ds_name)
+        ds_name = unique_name(ds_name)
+
+        # Code the sequence item dataset
+        code_item = code_dataset(
+            ds, ds_name, exclude_size, include_private, var_names=var_names
+        )
 
-        # Code the line to append the item to its parent sequence
-        lines.append(seq_var + ".append(" + ds_name + ")")
+        # Remove variable name from stored list, this dataset complete
+        var_names.pop()
+
+        # Code dataset creation and appending that to sequence, then the rest
+        # This keeps the logic close together, rather than after many items set
+        code_split = code_item.splitlines()
+        lines.append(code_split[0])  # "<ds_name> = Dataset()"
+        lines.append(f"{seq_var}.append({ds_name})")
+        lines.extend(code_split[1:])
+
+    # Remove sequence variable name we've used
+    var_names.pop()
 
     # Join the lines and return a single string
     return line_term.join(lines)
@@ -227,6 +261,7 @@ def code_dataset(
     exclude_size: Optional[int] = None,
     include_private: bool = False,
     is_file_meta: bool = False,
+    var_names: Optional[deque] = None
 ) -> str:
     """Return Python code for creating `ds`.
 
@@ -245,6 +280,8 @@ def code_dataset(
         data elements will be coded.
     is_file_meta : bool, optional
         ``True`` if `ds` contains file meta information elements.
+    var_names: deque, optional
+        Used internally to ensure unique variable names in nested sequences.
 
     Returns
     -------
@@ -252,8 +289,12 @@ def code_dataset(
         The codified dataset.
     """
 
+    if var_names is None:
+        var_names = deque()
     lines = []
+
     ds_class = " = FileMetaDataset()" if is_file_meta else " = Dataset()"
+
     lines.append(dataset_name + ds_class)
     for dataelem in ds:
         # If a private data element and flag says so, skip it and go to next
@@ -261,7 +302,8 @@ def code_dataset(
             continue
         # Otherwise code the line and add it to the lines list
         code_line = code_dataelem(
-            dataelem, dataset_name, exclude_size, include_private
+            dataelem, dataset_name, exclude_size, include_private,
+            var_names=var_names
         )
         lines.append(code_line)
         # Add blank line if just coded a sequence
@@ -270,6 +312,7 @@ def code_dataset(
     # If sequence was end of this dataset, remove the extra blank line
     if len(lines) and lines[-1] == "":
         lines.pop()
+
     # Join all the code lines and return them
     return line_term.join(lines)
 
@@ -313,8 +356,8 @@ def code_file_from_dataset(
 
     Parameters
     ----------
-    filename : str
-        Complete path and filename of a DICOM file to convert
+    ds : Dataset
+        A pydicom Dataset to convert
     exclude_size : Union[int,None]
         If not None, values longer than this (in bytes)
         will only have a commented string for a value,
@@ -336,6 +379,7 @@ def code_file_from_dataset(
     filename = ds.get("filename")
     identifier = f"DICOM file '{filename}'" if filename else "non-file dataset"
 
+    lines.append("# -*- coding: utf-8 -*-")
     lines.append(f"# Coded version of {identifier}")
     lines.append("# Produced by pydicom codify utility script")
 
@@ -385,7 +429,7 @@ def set_parser_arguments(
     parser.add_argument(
         "outfile",
         nargs="?",
-        type=argparse.FileType("w"),
+        type=argparse.FileType("w", encoding="UTF-8"),
         help=(
             "Filename to write Python code to, if not specified then code is "
             "written to stdout"


Test patch:
diff --git a/pydicom/tests/test_cli.py b/pydicom/tests/test_cli.py
--- a/pydicom/tests/test_cli.py
+++ b/pydicom/tests/test_cli.py
@@ -1,3 +1,4 @@
+# -*- coding: utf-8 -*-
 # Copyright 2020 pydicom authors. See LICENSE file for details.
 """Tests for command-line interface"""
 
@@ -138,6 +139,13 @@ def test_codify_data_element(self, capsys):
         with pytest.raises(NotImplementedError):
             main("codify pydicom::rtplan.dcm::RTPlanLabel".split())
 
+    def test_codify_UTF8(self, capsys):
+        """CLI `codify` command creates code with utf-8 characters"""
+        main(f"codify pydicom::chrFren.dcm".split())
+        out, _ = capsys.readouterr()
+        assert out.startswith("# -*- coding: utf-8 -*-")
+        assert "Buc^Jérôme" in out
+
     def test_help(self, capsys):
         """CLI `help` command gives expected output"""
         # With subcommand
diff --git a/pydicom/tests/test_util.py b/pydicom/tests/test_util.py
--- a/pydicom/tests/test_util.py
+++ b/pydicom/tests/test_util.py
@@ -5,12 +5,11 @@
 
 import pytest
 
-from pydicom import config, dcmread
+from pydicom import config, dcmread, Dataset, Sequence
 from pydicom import filereader
 from pydicom._private_dict import private_dictionaries
 from pydicom.data import get_testdata_file
 from pydicom.dataelem import DataElement
-from pydicom.dataset import Dataset
 from pydicom.tag import Tag
 from pydicom.uid import (
     ImplicitVRLittleEndian, ExplicitVRBigEndian, ExplicitVRLittleEndian
@@ -23,6 +22,7 @@
     default_name_filter,
     code_imports,
     code_dataelem,
+    code_dataset,
     main as codify_main,
 )
 from pydicom.util.dump import *
@@ -134,15 +134,34 @@ def test_code_sequence(self):
             "\n"
             "# Control Point Sequence: Control Point 1\n"
             "cp1 = Dataset()\n"
-            "cp1.PatientID = '1234'\n"
-            "cp_sequence.append(cp1)"
+            "cp_sequence.append(cp1)\n"
+            "cp1.PatientID = '1234'"
         )
 
         assert out == code_dataelem(elem)
 
-    def test_code_dataset(self):
-        """Test utils.codify.code_dataset"""
-        pass
+    def test_codify_recurring_keyword(self):
+        """Test utils.codify.code_dataset with same keyword nested"""
+        # Create fake Dataset with repeated DICOM keyword nested
+        # (0040, a730)  Content Sequence  1 item(s) ----
+        #    (0040, a040) Value Type                          CS: 'CODE'
+        #    (0040, a730)  Content Sequence  1 item(s) ----
+        #       (0040, a040) Value Type                          CS: 'CODE'
+
+        ds = Dataset()
+        ds.ContentSequence = seq1 = Sequence()
+        seq1.append(Dataset())
+        seq1[0].ValueType = "CODE"
+        seq1[0].ContentSequence = seq2 = Sequence()
+        seq2.append(Dataset())
+        seq2[0].ValueType = "CODE_1"
+        ds_code = code_dataset(ds)
+
+        # normal 1st use of var name
+        assert "content1.ValueType = 'CODE'" in ds_code
+
+        # Nested item of same name should have subscript
+        assert "content1_1.ValueType = 'CODE_1'" in ds_code
 
     def test_code_file(self, capsys):
         """Test utils.codify.code_file"""