Repository: kevoreilly/CAPEv2

Issue:
Summary reports cannot be generated if CAPA is enabled - jinja2.exceptions.TemplateRuntimeError: No filter named 'flare_capa_capability' found
# Expected Behavior

Users should be able to use CAPA and have still have summary reports generated.

# Current Behavior

What is the current behavior?

Creation of the summary HTML content (which is also used for summary report PDFs) fail

# Failure Information (for bugs)

```text
May 16 00:26:53 sandbox python3[65267]: ERROR:lib.cuckoo.core.plugins:Failed to run the reporting module "ReportHTMLSummary": No filter named 'flare_capa_capability' found.
May 16 00:26:53 sandbox python3[65267]: Traceback (most recent call last):
May 16 00:26:53 sandbox python3[65267]:   File "/opt/CAPEv2/utils/../lib/cuckoo/core/plugins.py", line 712, in process
May 16 00:26:53 sandbox python3[65267]:     current.run(self.results)
May 16 00:26:53 sandbox python3[65267]:   File "/opt/CAPEv2/utils/../modules/reporting/reporthtmlsummary.py", line 94, in run
May 16 00:26:53 sandbox python3[65267]:     html = tpl.render({"results": results, "summary_report": True})
May 16 00:26:53 sandbox python3[65267]:   File "/home/cape/.cache/pypoetry/virtualenvs/capev2-t2x27zRb-py3.10/lib/python3.10/site-packages/jinja2/environment.py", line 1301, in render
May 16 00:26:53 sandbox python3[65267]:     self.environment.handle_exception()
May 16 00:26:53 sandbox python3[65267]:   File "/home/cape/.cache/pypoetry/virtualenvs/capev2-t2x27zRb-py3.10/lib/python3.10/site-packages/jinja2/environment.py", line 936, in handle_except>
May 16 00:26:53 sandbox python3[65267]:     raise rewrite_traceback_stack(source=source)
May 16 00:26:53 sandbox python3[65267]:   File "/opt/CAPEv2/data/html/report.html", line 1, in top-level template code
May 16 00:26:53 sandbox python3[65267]:     {% extends "base-report.html" %}
May 16 00:26:53 sandbox python3[65267]:   File "/opt/CAPEv2/data/html/base-report.html", line 63, in top-level template code
May 16 00:26:53 sandbox python3[65267]:     {% block content %}{% endblock %}
May 16 00:26:53 sandbox python3[65267]:   File "/opt/CAPEv2/data/html/report.html", line 6, in block 'content'
May 16 00:26:53 sandbox python3[65267]:     {% include "sections/file.html" %}
May 16 00:26:53 sandbox python3[65267]:   File "/opt/CAPEv2/data/html/sections/file.html", line 18, in top-level template code
May 16 00:26:53 sandbox python3[65267]:     {% include "generic/_file_info.html" %}
May 16 00:26:53 sandbox python3[65267]:   File "/opt/CAPEv2/data/html/generic/_file_info.html", line 280, in top-level template code
May 16 00:26:53 sandbox python3[65267]:     {{file.flare_capa|flare_capa_capability}}
May 16 00:26:53 sandbox python3[65267]: jinja2.exceptions.TemplateRuntimeError: No filter named 'flare_capa_capability' found.
May 16 00:26:53 sandbox python3[65267]: 2023-05-16 00:26:53,003 [Task 1] [lib.cuckoo.core.plugins] ERROR: Failed to run the reporting module "ReportHTMLSummary": No filter named 'flare_capa>
May 16 00:26:53 sandbox python3[65267]: Traceback (most recent call last):
May 16 00:26:53 sandbox python3[65267]:   File "/opt/CAPEv2/utils/../lib/cuckoo/core/plugins.py", line 712, in process
May 16 00:26:53 sandbox python3[65267]:     current.run(self.results)
May 16 00:26:53 sandbox python3[65267]:   File "/opt/CAPEv2/utils/../modules/reporting/reporthtmlsummary.py", line 94, in run
May 16 00:26:53 sandbox python3[65267]:     html = tpl.render({"results": results, "summary_report": True})
May 16 00:26:53 sandbox python3[65267]:   File "/home/cape/.cache/pypoetry/virtualenvs/capev2-t2x27zRb-py3.10/lib/python3.10/site-packages/jinja2/environment.py", line 1301, in render
May 16 00:26:53 sandbox python3[65267]:     self.environment.handle_exception()
May 16 00:26:53 sandbox python3[65267]:   File "/home/cape/.cache/pypoetry/virtualenvs/capev2-t2x27zRb-py3.10/lib/python3.10/site-packages/jinja2/environment.py", line 936, in handle_except>
May 16 00:26:53 sandbox python3[65267]:     raise rewrite_traceback_stack(source=source)
May 16 00:26:53 sandbox python3[65267]:   File "/opt/CAPEv2/data/html/report.html", line 1, in top-level template code
May 16 00:26:53 sandbox python3[65267]:     {% extends "base-report.html" %}
May 16 00:26:53 sandbox python3[65267]:   File "/opt/CAPEv2/data/html/base-report.html", line 63, in top-level template code
May 16 00:26:53 sandbox python3[65267]:     {% block content %}{% endblock %}
May 16 00:26:53 sandbox python3[65267]:   File "/opt/CAPEv2/data/html/report.html", line 6, in block 'content'
May 16 00:26:53 sandbox python3[65267]:     {% include "sections/file.html" %}
May 16 00:26:53 sandbox python3[65267]:   File "/opt/CAPEv2/data/html/sections/file.html", line 18, in top-level template code
May 16 00:26:53 sandbox python3[65267]:     {% include "generic/_file_info.html" %}
May 16 00:26:53 sandbox python3[65267]:   File "/opt/CAPEv2/data/html/generic/_file_info.html", line 280, in top-level template code
May 16 00:26:53 sandbox python3[65267]:     {{file.flare_capa|flare_capa_capability}}
May 16 00:26:53 sandbox python3[65267]: jinja2.exceptions.TemplateRuntimeError: No filter named 'flare_capa_capability' found.
May 16 00:26:53 sandbox python3[65267]: 2023-05-16 00:26:53,003 [Task 1] [lib.cuckoo.core.plugins] ERROR: Failed to run the reporting module "ReportHTMLSummary": No filter named 'flare_capa>
May 16 00:26:53 sandbox python3[65267]: Traceback (most recent call last):
May 16 00:26:53 sandbox python3[65267]:   File "/opt/CAPEv2/utils/../lib/cuckoo/core/plugins.py", line 712, in process
May 16 00:26:53 sandbox python3[65267]:     current.run(self.results)
May 16 00:26:53 sandbox python3[65267]:   File "/opt/CAPEv2/utils/../modules/reporting/reporthtmlsummary.py", line 94, in run
May 16 00:26:53 sandbox python3[65267]:     html = tpl.render({"results": results, "summary_report": True})
May 16 00:26:53 sandbox python3[65267]:   File "/home/cape/.cache/pypoetry/virtualenvs/capev2-t2x27zRb-py3.10/lib/python3.10/site-packages/jinja2/environment.py", line 1301, in render
May 16 00:26:53 sandbox python3[65267]:     self.environment.handle_exception()
May 16 00:26:53 sandbox python3[65267]:   File "/home/cape/.cache/pypoetry/virtualenvs/capev2-t2x27zRb-py3.10/lib/python3.10/site-packages/jinja2/environment.py", line 936, in handle_except>
May 16 00:26:53 sandbox python3[65267]:     raise rewrite_traceback_stack(source=source)
May 16 00:26:53 sandbox python3[65267]:   File "/opt/CAPEv2/data/html/report.html", line 1, in top-level template code
May 16 00:26:53 sandbox python3[65267]:     {% extends "base-report.html" %}
May 16 00:26:53 sandbox python3[65267]:   File "/opt/CAPEv2/data/html/base-report.html", line 63, in top-level template code
May 16 00:26:53 sandbox python3[65267]:     {% block content %}{% endblock %}
May 16 00:26:53 sandbox python3[65267]:   File "/opt/CAPEv2/data/html/report.html", line 6, in block 'content'
May 16 00:26:53 sandbox python3[65267]:     {% include "sections/file.html" %}
May 16 00:26:53 sandbox python3[65267]:   File "/opt/CAPEv2/data/html/sections/file.html", line 18, in top-level template code
May 16 00:26:53 sandbox python3[65267]:     {% include "generic/_file_info.html" %}
May 16 00:26:53 sandbox python3[65267]:   File "/opt/CAPEv2/data/html/generic/_file_info.html", line 280, in top-level template code
May 16 00:26:53 sandbox python3[65267]:     {{file.flare_capa|flare_capa_capability}}
May 16 00:26:53 sandbox python3[65267]: jinja2.exceptions.TemplateRuntimeError: No filter named 'flare_capa_capability' found.
May 16 00:26:53 sandbox python3[65267]: 2023-05-16 00:26:53,003 [Task 1] [lib.cuckoo.core.plugins] ERROR: Failed to run the reporting module "ReportHTMLSummary": No filter named 'flare_capa>
May 16 00:26:53 sandbox python3[65267]: Traceback (most recent call last):
May 16 00:26:53 sandbox python3[65267]:   File "/opt/CAPEv2/utils/../lib/cuckoo/core/plugins.py", line 712, in process
May 16 00:26:53 sandbox python3[65267]:     current.run(self.results)
May 16 00:26:53 sandbox python3[65267]:   File "/opt/CAPEv2/utils/../modules/reporting/reporthtmlsummary.py", line 94, in run
May 16 00:26:53 sandbox python3[65267]:     html = tpl.render({"results": results, "summary_report": True})
May 16 00:26:53 sandbox python3[65267]:   File "/home/cape/.cache/pypoetry/virtualenvs/capev2-t2x27zRb-py3.10/lib/python3.10/site-packages/jinja2/environment.py", line 1301, in render
May 16 00:26:53 sandbox python3[65267]:     self.environment.handle_exception()
May 16 00:26:53 sandbox python3[65267]:   File "/home/cape/.cache/pypoetry/virtualenvs/capev2-t2x27zRb-py3.10/lib/python3.10/site-packages/jinja2/environment.py", line 936, in handle_except>
May 16 00:26:53 sandbox python3[65267]:     raise rewrite_traceback_stack(source=source)
May 16 00:26:53 sandbox python3[65267]:   File "/opt/CAPEv2/data/html/report.html", line 1, in top-level template code
May 16 00:26:53 sandbox python3[65267]:     {% extends "base-report.html" %}
May 16 00:26:53 sandbox python3[65267]:   File "/opt/CAPEv2/data/html/base-report.html", line 63, in top-level template code
May 16 00:26:53 sandbox python3[65267]:     {% block content %}{% endblock %}
May 16 00:26:53 sandbox python3[65267]:   File "/opt/CAPEv2/data/html/report.html", line 6, in block 'content'
May 16 00:26:53 sandbox python3[65267]:     {% include "sections/file.html" %}
May 16 00:26:53 sandbox python3[65267]:   File "/opt/CAPEv2/data/html/sections/file.html", line 18, in top-level template code
May 16 00:26:53 sandbox python3[65267]:     {% include "generic/_file_info.html" %}
May 16 00:26:53 sandbox python3[65267]:   File "/opt/CAPEv2/data/html/generic/_file_info.html", line 280, in top-level template code
May 16 00:26:53 sandbox python3[65267]:     {{file.flare_capa|flare_capa_capability}}
May 16 00:26:53 sandbox python3[65267]: jinja2.exceptions.TemplateRuntimeError: No filter named 'flare_capa_capability' found.
May 16 00:26:53 sandbox python3[65267]: 2023-05-16 00:26:53,003 [Task 1] [lib.cuckoo.core.plugins] ERROR: Failed to run the reporting module "ReportHTMLSummary": No filter named 'flare_capa>
May 16 00:26:53 sandbox python3[65267]: Traceback (most recent call last):
May 16 00:26:53 sandbox python3[65267]:   File "/opt/CAPEv2/utils/../lib/cuckoo/core/plugins.py", line 712, in process
May 16 00:26:53 sandbox python3[65267]:     current.run(self.results)
May 16 00:26:53 sandbox python3[65267]:   File "/opt/CAPEv2/utils/../modules/reporting/reporthtmlsummary.py", line 94, in run
May 16 00:26:53 sandbox python3[65267]:     html = tpl.render({"results": results, "summary_report": True})
May 16 00:26:53 sandbox python3[65267]:   File "/home/cape/.cache/pypoetry/virtualenvs/capev2-t2x27zRb-py3.10/lib/python3.10/site-packages/jinja2/environment.py", line 1301, in render
May 16 00:26:53 sandbox python3[65267]:     self.environment.handle_exception()
May 16 00:26:53 sandbox python3[65267]:   File "/home/cape/.cache/pypoetry/virtualenvs/capev2-t2x27zRb-py3.10/lib/python3.10/site-packages/jinja2/environment.py", line 936, in handle_except>
May 16 00:26:53 sandbox python3[65267]:     raise rewrite_traceback_stack(source=source)
May 16 00:26:53 sandbox python3[65267]:   File "/opt/CAPEv2/data/html/report.html", line 1, in top-level template code
May 16 00:26:53 sandbox python3[65267]:     {% extends "base-report.html" %}
May 16 00:26:53 sandbox python3[65267]:   File "/opt/CAPEv2/data/html/base-report.html", line 63, in top-level template code
May 16 00:26:53 sandbox python3[65267]:     {% block content %}{% endblock %}
May 16 00:26:53 sandbox python3[65267]:   File "/opt/CAPEv2/data/html/report.html", line 6, in block 'content'
May 16 00:26:53 sandbox python3[65267]:     {% include "sections/file.html" %}
May 16 00:26:53 sandbox python3[65267]:   File "/opt/CAPEv2/data/html/sections/file.html", line 18, in top-level template code
May 16 00:26:53 sandbox python3[65267]:     {% include "generic/_file_info.html" %}
May 16 00:26:53 sandbox python3[65267]:   File "/opt/CAPEv2/data/html/generic/_file_info.html", line 280, in top-level template code
May 16 00:26:53 sandbox python3[65267]:     {{file.flare_capa|flare_capa_capability}}
May 16 00:26:53 sandbox python3[65267]: jinja2.exceptions.TemplateRuntimeError: No filter named 'flare_capa_capability' found.
May 16 00:26:53 sandbox python3[65267]: WARNING:lib.cuckoo.core.plugins:The reporting module "ReportPDF" returned the following error: Unable to open summary HTML report to convert to PDF: >
May 16 00:26:53 sandbox python3[65267]: 2023-05-16 00:26:53,021 [Task 1] [lib.cuckoo.core.plugins] WARNING: The reporting module "ReportPDF" returned the following error: Unable to open sum>
May 16 00:26:53 sandbox python3[65267]: 2023-05-16 00:26:53,021 [Task 1] [lib.cuckoo.core.plugins] WARNING: The reporting module "ReportPDF" returned the following error: Unable to open sum>
May 16 00:26:53 sandbox python3[65256]: INFO:root:Reports generation completed
May 16 00:26:53 sandbox python3[65256]: 2023-05-16 00:26:53,458 [root] INFO: Reports generation completed
```

## Steps to Reproduce

Please provide detailed steps for reproducing the issue.

1. Enable CAPA in `processing.conf`
2. Enable `reporthtmlsummary` and `reportpdf in reporting.conf`
3. Upload a PE sample for analysis

## Context

| Question         | Answer
|-----------------|--------------------
| Git commit      | 9296021e104c2a53da49195bdacb1bd83626539d
| OS version       | Ubuntu 22.04

## Failure Logs

Please include any relevant log snippets or files here.



Discussion:
it works just fine here. module is there, so i guess is something wrong on your side

https://github.com/kevoreilly/CAPEv2/blob/master/modules/reporting/reporthtmlsummary.py#L19
https://github.com/kevoreilly/CAPEv2/blob/master/modules/reporting/reporthtmlsummary.py#LL85C44-L85C59
Weird. I'm trying to think of what that could be. What version of Jinja2 are you using?
https://github.com/kevoreilly/CAPEv2/blob/master/pyproject.toml#L42, that sound more as import error than jinja error, try import by hand
Manual imports work fine. Also looking at the code again, that import isn't part of the try/except block, so I wouldn't be able to generate any report at all if it was an import issue. Weird. I asked about the version what because I haven't  seen ^ used in a version specification before, and couldn't find where that is defined after some Googling (for what it's worth 3.1.2 is installed in my environment. Also, are you running the latest commit, like I am? Just trying to figure out what could be different with my instance that would cause this...
^ means start with

Yes but im on staging branch, which is kinda the same, we merge ir to
máster each wednesday

El mar, 16 may 2023 17:03, Sean Whalen ***@***.***> escribió:

> Manual imports work fine. Also looking at the code again, that import
> isn't part of the try/except block, so I wouldn't be able to generate any
> report at all if it was an import issue. Weird. I asked about the version
> what because I haven't seen ^ used in a version specification before, and
> couldn't find where that is defined after some Googling (for what it's
> worth 3.1.2 is installed in my environment. Also, are you running the
> latest commit, like I am? Just trying to figure out what could be different
> with my instance that would cause this...
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/kevoreilly/CAPEv2/issues/1523#issuecomment-1549852964>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AAOFH36PIFZW77RMYN2Z5NLXGOJNTANCNFSM6AAAAAAYC5D3FA>
> .
> You are receiving this because you commented.Message ID:
> ***@***.***>
>

Hmm. I'd like to tun the `run` method in `reporthtmlsummary.py` through a debugger. What's the best way to get the `results`  `dict``that is expected as input?
if you go with debugger put breakpoint and you will see it as self.results or just results or report i don't remember, just use `dir()`
This was my first time using `pdb` directly, instead of using an IDE, so I'm probably doing something wrong here. Any ideas?

What I did:

1. Stopped my `cape-processing` service
2. Added `breakpoint()` as line 91 of `reporthtmlsummary.py`
3. Executed `sudo -u cape python3 -m poetry run python process.py -p7 auto -pt 900```
4. Submitted a sample

What happened

```python
 /opt/CAPEv2/modules/reporting/reporthtmlsummary.py(92)run()
-> env.loader = FileSystemLoader(os.path.join(CUCKOO_ROOT, "data", "html"))
(Pdb) 
ERROR:lib.cuckoo.core.plugins:Failed to run the reporting module "ReportHTMLSummary": 
Traceback (most recent call last):
  File "/opt/CAPEv2/utils/../lib/cuckoo/core/plugins.py", line 712, in process
    current.run(self.results)
  File "/opt/CAPEv2/utils/../modules/reporting/reporthtmlsummary.py", line 92, in run
    env.loader = FileSystemLoader(os.path.join(CUCKOO_ROOT, "data", "html"))
  File "/opt/CAPEv2/utils/../modules/reporting/reporthtmlsummary.py", line 92, in run
    env.loader = FileSystemLoader(os.path.join(CUCKOO_ROOT, "data", "html"))
  File "/usr/lib/python3.10/bdb.py", line 90, in trace_dispatch
    return self.dispatch_line(frame)
  File "/usr/lib/python3.10/bdb.py", line 115, in dispatch_line
    if self.quitting: raise BdbQuit
bdb.BdbQuit
2023-05-17 18:15:54,415 [Task 7] [lib.cuckoo.core.plugins] ERROR: Failed to run the reporting module "ReportHTMLSummary": 
Traceback (most recent call last):
  File "/opt/CAPEv2/utils/../lib/cuckoo/core/plugins.py", line 712, in process
    current.run(self.results)
  File "/opt/CAPEv2/utils/../modules/reporting/reporthtmlsummary.py", line 92, in run
    env.loader = FileSystemLoader(os.path.join(CUCKOO_ROOT, "data", "html"))
  File "/opt/CAPEv2/utils/../modules/reporting/reporthtmlsummary.py", line 92, in run
    env.loader = FileSystemLoader(os.path.join(CUCKOO_ROOT, "data", "html"))
  File "/usr/lib/python3.10/bdb.py", line 90, in trace_dispatch
    return self.dispatch_line(frame)
  File "/usr/lib/python3.10/bdb.py", line 115, in dispatch_line
    if self.quitting: raise BdbQuit
bdb.BdbQuit
2023-05-17 18:15:54,415 [Task 7] [lib.cuckoo.core.plugins] ERROR: Failed to run the reporting module "ReportHTMLSummary": 
Traceback (most recent call last):
  File "/opt/CAPEv2/utils/../lib/cuckoo/core/plugins.py", line 712, in process
    current.run(self.results)
  File "/opt/CAPEv2/utils/../modules/reporting/reporthtmlsummary.py", line 92, in run
    env.loader = FileSystemLoader(os.path.join(CUCKOO_ROOT, "data", "html"))
  File "/opt/CAPEv2/utils/../modules/reporting/reporthtmlsummary.py", line 92, in run
    env.loader = FileSystemLoader(os.path.join(CUCKOO_ROOT, "data", "html"))
  File "/usr/lib/python3.10/bdb.py", line 90, in trace_dispatch
    return self.dispatch_line(frame)
  File "/usr/lib/python3.10/bdb.py", line 115, in dispatch_line
    if self.quitting: raise BdbQuit
bdb.BdbQuit
```
that not related at all to your problem, instead of `breakpoint()` i personally use oneliner in most of the cases `import code;code.interact(local=dict(locals(), **globals()))` will give you a bit easier to control over than breakpoint. i will try to test it again but till july im having almost no spare time
ok i was able to enable capa and summaryhtmlreport and with reprocess i don't have any problem. i have verified jinja version, capa is 5.1.0. 
The only other potential difference I can think of is Ubuntu and Python version. I'm running Ubuntu 22.04, which uses Python 3.10. You?
The same, python 3.10.6

El vie, 19 may 2023 3:16, Sean Whalen ***@***.***> escribió:

> The only other potential difference I can think of is Ubuntu and Python
> version. I'm running Ubuntu 22.04, which uses Python 3.10. You?
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/kevoreilly/CAPEv2/issues/1523#issuecomment-1553879627>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AAOFH3YHDG7CXSII2EGLC4TXG3CXRANCNFSM6AAAAAAYC5D3FA>
> .
> You are receiving this because you commented.Message ID:
> ***@***.***>
>

If you need that reporte, just remove html data with that problematic code
in data/html/

El vie, 19 may 2023 8:10, Andriy Brukhovetskyy ***@***.***>
escribió:

> The same, python 3.10.6
>
> El vie, 19 may 2023 3:16, Sean Whalen ***@***.***> escribió:
>
>> The only other potential difference I can think of is Ubuntu and Python
>> version. I'm running Ubuntu 22.04, which uses Python 3.10. You?
>>
>> —
>> Reply to this email directly, view it on GitHub
>> <https://github.com/kevoreilly/CAPEv2/issues/1523#issuecomment-1553879627>,
>> or unsubscribe
>> <https://github.com/notifications/unsubscribe-auth/AAOFH3YHDG7CXSII2EGLC4TXG3CXRANCNFSM6AAAAAAYC5D3FA>
>> .
>> You are receiving this because you commented.Message ID:
>> ***@***.***>
>>
>


Resolving patch:
diff --git a/modules/reporting/reporthtml.py b/modules/reporting/reporthtml.py
--- a/modules/reporting/reporthtml.py
+++ b/modules/reporting/reporthtml.py
@@ -69,7 +69,7 @@ def run(self, results):
                 "dict2list": dict2list,
                 "parentfixup": parentfixup,
                 "malware_config": malware_config,
-                "flare_capa_capabilities": flare_capa_capabilities,
+                "flare_capa_capability": flare_capa_capabilities,
                 "flare_capa_attck": flare_capa_attck,
                 "flare_capa_mbc": flare_capa_mbc,
                 "datefmt": datefmt,
diff --git a/modules/reporting/reporthtmlsummary.py b/modules/reporting/reporthtmlsummary.py
--- a/modules/reporting/reporthtmlsummary.py
+++ b/modules/reporting/reporthtmlsummary.py
@@ -82,7 +82,7 @@ def run(self, results):
                 "dict2list": dict2list,
                 "parentfixup": parentfixup,
                 "malware_config": malware_config,
-                "flare_capa_capabilities": flare_capa_capabilities,
+                "flare_capa_capability": flare_capa_capabilities,
                 "flare_capa_attck": flare_capa_attck,
                 "flare_capa_mbc": flare_capa_mbc,
                 "datefmt": datefmt,


Test patch: