internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: rpm-software-management/dnf

Issue:
Bug on almalinux 9
Hi,

I have these messages on almalinux 9/ Rocky Linux 9


[mickael@srvalmalinux ~]$ sudo  /usr/bin/dnf-automatic /etc/dnf/automatic.conf --timer
Last metadata expiration check: 2:45:52 ago on Fri Mar 10 14:38:38 2023.
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
Traceback (most recent call last):
  File "/usr/bin/dnf-automatic", line 36, in <module>
    sys.exit(dnf.automatic.main.main(sys.argv[1:]))
  File "/usr/lib/python3.9/site-packages/dnf/automatic/main.py", line 342, in main
    emitters.commit()
  File "/usr/lib/python3.9/site-packages/dnf/util.py", line 526, in fn
    return list(map(call_what, self))
  File "/usr/lib/python3.9/site-packages/dnf/util.py", line 525, in call_what
    return method(*args, **kwargs)
  File "/usr/lib/python3.9/site-packages/dnf/automatic/emitter.py", line 106, in commit
    smtp = smtplib.SMTP(self._conf.email_host, timeout=300)
  File "/usr/lib64/python3.9/smtplib.py", line 255, in __init__
    (code, msg) = self.connect(host, port)
  File "/usr/lib64/python3.9/smtplib.py", line 341, in connect
    self.sock = self._get_socket(host, port, self.timeout)
  File "/usr/lib64/python3.9/smtplib.py", line 312, in _get_socket
    return socket.create_connection((host, port), timeout,
  File "/usr/lib64/python3.9/socket.py", line 844, in create_connection
    raise err
  File "/usr/lib64/python3.9/socket.py", line 832, in create_connection
    sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused



Discussion:

Resolving patch:
diff --git a/dnf/automatic/emitter.py b/dnf/automatic/emitter.py
--- a/dnf/automatic/emitter.py
+++ b/dnf/automatic/emitter.py
@@ -106,7 +106,7 @@ def commit(self):
             smtp = smtplib.SMTP(self._conf.email_host, timeout=300)
             smtp.sendmail(email_from, email_to, message.as_string())
             smtp.close()
-        except smtplib.SMTPException as exc:
+        except OSError as exc:
             msg = _("Failed to send an email via '%s': %s") % (
                 self._conf.email_host, exc)
             logger.error(msg)


Test patch:
Repository: rpm-software-management/dnf

Issue:
`email_port` option supported but not used
It looks like the `email_port` option, while parsed in [dnf/automatic/main.py:200](https://github.com/rpm-software-management/dnf/blob/master/dnf/automatic/main.py#L200), is not actually used anywhere.


Discussion:

Resolving patch:
diff --git a/dnf/automatic/emitter.py b/dnf/automatic/emitter.py
--- a/dnf/automatic/emitter.py
+++ b/dnf/automatic/emitter.py
@@ -95,6 +95,7 @@ def commit(self):
         message.set_charset('utf-8')
         email_from = self._conf.email_from
         email_to = self._conf.email_to
+        email_port = self._conf.email_port
         message['Date'] = email.utils.formatdate()
         message['From'] = email_from
         message['Subject'] = subj
@@ -103,7 +104,7 @@ def commit(self):
 
         # Send the email
         try:
-            smtp = smtplib.SMTP(self._conf.email_host, timeout=300)
+            smtp = smtplib.SMTP(self._conf.email_host, self._conf.email_port, timeout=300)
             smtp.sendmail(email_from, email_to, message.as_string())
             smtp.close()
         except OSError as exc:


Test patch:
Repository: OpenCTI-Platform/connectors

Issue:
[CVE] Download link to variable
## Description

Set the download CVE link to variable, because otherwise the tool can hardly be used offline. Offline we can host the CVEs on a link that is not : "https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-"

## Reproducible Steps

https://github.com/OpenCTI-Platform/connectors/blame/9d47ffdad1c2a7fbdd709565d5c3f670693b148f/cve/src/cve.py#L103

## Expected Output

Url as a variable in the .yml

## Actual Output

Permanent link : "https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-"



Discussion:

Resolving patch:
diff --git a/cve/src/cve.py b/cve/src/cve.py
--- a/cve/src/cve.py
+++ b/cve/src/cve.py
@@ -29,6 +29,9 @@ def __init__(self):
         self.cve_nvd_data_feed = get_config_variable(
             "CVE_NVD_DATA_FEED", ["cve", "nvd_data_feed"], config
         )
+        self.cve_history_data_feed = get_config_variable(
+            "CVE_HISTORY_DATA_FEED", ["cve", "history_data_feed"], config
+        )
         self.cve_interval = get_config_variable(
             "CVE_INTERVAL", ["cve", "interval"], config, True
         )
@@ -97,12 +100,10 @@ def run(self):
                     # If import history and never run
                     if last_run is None and self.cve_import_history:
                         now = datetime.now()
-                        years = list(range(2002, now.year))
+                        years = list(range(2002, now.year+1))
                         for year in years:
                             self.convert_and_send(
-                                "https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-"
-                                + str(year)
-                                + ".json.gz"
+                                f"{self.cve_history_data_feed}nvdcve-1.1-{year}.json.gz"
                             )
 
                     # Store the current timestamp as a last run


Test patch:
Repository: OpenCTI-Platform/connectors

Issue:
[cybercrime-tracker] connector fails due to unchecked None
## Description

Cybercrime tracker connector fails while determining if the last run interval has been exceeded.

## Environment

1. OS (where OpenCTI server runs): Debian Buster 10.7
2. OpenCTI version: 4.0.x
3. OpenCTI client: pycti
4. Other environment details:


## Reproducible Steps

Steps to create the smallest reproducible scenario:
1. Setup the connector in accordance its the README (docker)
2. `docker-compose up`
3. See "ERROR" logs

```
Attaching to cybercrime-tracker_connector-cybercrimetracker_1
connector-cybercrimetracker_1  | INFO:root:Listing Threat-Actors with filters null.
connector-cybercrimetracker_1  | INFO:root:Connector registered with ID:cybercrime
connector-cybercrimetracker_1  | INFO:root:Starting ping alive thread
connector-cybercrimetracker_1  | INFO:root:Fetching data CYBERCRIME-TRACKER.NET...
connector-cybercrimetracker_1  | INFO:root:Listing Marking-Definitions with filters [{"key": "definition", "values": "TLP:WHITE"}].
connector-cybercrimetracker_1  | INFO:root:Connector last run: 2020-12-21 05:57:36
connector-cybercrimetracker_1  | ERROR:root:'>' not supported between instances of 'int' and 'NoneType'
```
This error seems to occur when [determining if it's time to run](https://github.com/OpenCTI-Platform/connectors/blob/master/cybercrime-tracker/src/cybercrime-tracker.py#L163).

## Expected Output

I expected the Data page to display the number of messages queued from this source.

## Actual Output

0 messages queued


Discussion:

Resolving patch:
diff --git a/cybercrime-tracker/src/cybercrime-tracker.py b/cybercrime-tracker/src/cybercrime-tracker.py
--- a/cybercrime-tracker/src/cybercrime-tracker.py
+++ b/cybercrime-tracker/src/cybercrime-tracker.py
@@ -56,7 +56,7 @@ def __init__(self):
             config,
         )
         self.interval = get_config_variable(
-            "CYBERCRIMETRACKER_INTERVAL",
+            "CYBERCRIME_TRACKER_INTERVAL",
             ["cybercrime-tracker", "interval"],
             config,
             isNumber=True,


Test patch:
Repository: OpenCTI-Platform/connectors

Issue:
MISP only imports if there is galaxy tag from mitre-attack is added
## Description

I would like OpenCTI to also import events that do not have galaxy tags from the mitre-attack galaxy

## Environment

1. OS (where OpenCTI server runs): Debian 10
2. OpenCTI version: 4.3.4
3. OpenCTI client: docker
4. Other environment details: OpenCTI + MISP with default env
misp connector:
- MISP_CREATE_REPORTS=True # Required, create report for MISP event
- MISP_IMPORT_TAGS=opencti:import

## Reproducible Steps

1. Create MISP event with title, opencti:import tag, observables
2. Either don't add a galaxy tag, or one that isn't from mitre-attack
3. Publish

The event will not goto OpenCTI
If you reproduce the same steps but add e.g. attack patterns from mitre-attack it will get imported.
Is this the intended behaviour?

## Expected Output

OpenCTI imports all events


Discussion:

Resolving patch:
diff --git a/misp/src/misp.py b/misp/src/misp.py
--- a/misp/src/misp.py
+++ b/misp/src/misp.py
@@ -4,7 +4,6 @@
 import json
 
 from datetime import datetime
-from dateutil.parser import parse
 from pymisp import ExpandedPyMISP
 from stix2 import (
     Bundle,
@@ -42,6 +41,7 @@
         "transform": {"operation": "remove_string", "value": "AS"},
     },
     "mac-addr": {"type": "mac-addr", "path": ["value"]},
+    "hostname": {"type": "x-opencti-hostname", "path": ["value"]},
     "domain": {"type": "domain-name", "path": ["value"]},
     "ipv4-addr": {"type": "ipv4-addr", "path": ["value"]},
     "ipv6-addr": {"type": "ipv6-addr", "path": ["value"]},
@@ -488,12 +488,14 @@ def process_events(self, work_id, events):
                 bundle_objects.append(object_relationship)
 
             # Create the report if needed
-            if self.misp_create_report and len(object_refs) > 0:
+            if self.misp_create_report:
                 report = Report(
                     id="report--" + event["Event"]["uuid"],
                     name=event["Event"]["info"],
                     description=event["Event"]["info"],
-                    published=parse(event["Event"]["date"]),
+                    published=datetime.utcfromtimestamp(
+                        int(event["Event"]["timestamp"])
+                    ),
                     report_types=[self.misp_report_type],
                     created_by_ref=author,
                     object_marking_refs=event_markings,


Test patch:
Repository: OpenCTI-Platform/connectors

Issue:
Riskiq Connector throwing errors
## Description

RiskIQ connector is not working as expected with the correct credentials defined.

## Environment

1. OS - Ubuntu 
2. OpenCTI version: 5.1.3

## Riskiq Connector Logs:
INFO:root:Listing Threat-Actors with filters null.,
INFO:root:Connector registered with ID: c455a3a4-cc8f-4133-9f8d-4098fa984de8,
INFO:root:Starting ping alive thread,
INFO:riskiq.client:URL: https://api.riskiq.net/pt/v2,
INFO:root:Starting RiskIQ connector...,
INFO:root:Running RiskIQ connector...,
INFO:root:Connector interval sec: 60,
INFO:root:[RiskIQ] loaded state: {},
INFO:root:RiskIQ connector clean run,
INFO:root:Initiate work for c455a3a4-cc8f-4133-9f8d-4098fa984de8,
INFO:root:[RiskIQ] workid opencti-work--2c314a8c-484e-4a68-9b31-bb782b3b22ed initiated,
INFO:root:[RiskIQ] last run: None,
**ERROR:root:Parser must be a string or character stream, not NoneType **

![image](https://user-images.githubusercontent.com/92973503/151704741-cd22d3ec-4963-4f7f-becd-d4930a9d0798.png)

++Config File++

  connector-riskiq:
    image: opencti/connector-riskiq:5.1.3
    environment:
      - OPENCTI_URL=http://opencti:8080
      - OPENCTI_TOKEN=c9dc7053-6bdf-44ca-9dfd-c0e3ff249eb8
      - CONNECTOR_ID=c455a3a4-cc8f-4133-9f8d-4098fa984de8
      - CONNECTOR_TYPE=EXTERNAL_IMPORT
      - CONNECTOR_NAME=RISKIQ
      - CONNECTOR_SCOPE=riskiq
      - CONNECTOR_CONFIDENCE_LEVEL=15 # From 0 (Unknown) to 100 (Fully trusted)
      - CONNECTOR_LOG_LEVEL=info
      - RISKIQ_BASE_URL=https://api.riskiq.net/pt/v2
      - RISKIQ_USER=xxx@testing.com
      - RISKIQ_PASSWORD=xxxxxxx
      - RISKIQ_INTERVAL_SEC=86400
    restart: always


It was working before, after a reboot the riskiq connector started logging the above error as "ERROR:root:Parser must be a string or character stream, not NoneType".

Please help to fix the same.

Thanks



Discussion:
@axelfahy : Any idea where that error could come from?
I can reproduce, it seems the API is not providing the `publishedDate` anymore. I'll fix it.

Resolving patch:
diff --git a/external-import/riskiq/src/riskiq/article_importer.py b/external-import/riskiq/src/riskiq/article_importer.py
--- a/external-import/riskiq/src/riskiq/article_importer.py
+++ b/external-import/riskiq/src/riskiq/article_importer.py
@@ -222,8 +222,14 @@ def _process_indicator(self, indicator: Indicator) -> list[_Observable]:
     def run(self, work_id: str, state: Mapping[str, Any]) -> Mapping[str, Any]:
         """Run the importation of the article."""
         self.work_id = work_id
-        published = parser.parse(self.article["publishedDate"])
         created = parser.parse(self.article["createdDate"])
+        # RisIQ API does not always provide the `publishedDate`.
+        # If it does not exist, take the value of the `createdDate` instead.
+        published = (
+            parser.parse(self.article["publishedDate"])
+            if self.article["publishedDate"] is not None
+            else created
+        )
 
         indicators = itertools.chain(
             *[


Test patch:
Repository: OpenCTI-Platform/connectors

Issue:
[VXVault] Connector imports garbage data (HTML tags and blank events)
## Description

The VXVault connector is importing garbage data as indicators and observables, including events like `<pre>`, </pre>` and blank entities. 

## Environment

1. OS (where OpenCTI server runs): Docker on Ubuntu 20 
2. OpenCTI version: 5.5.2
3. OpenCTI client: connectors/python

## Expected Output
Connector should download the URLs from the VXVault threat feed here: http://vxvault.net/URL_List.php and import them into OpenCTI as Observables and Indicators.

## Actual Output
The connector does import the URLs, but it fails to parse the page correctly and imports some of the HTML tags as well by accident, leading to garbage data like Indicators with names and values as just `<pre>`

## Additional information
I discussed this and a couple of other related issues with the VXVault connector + data with Samuel Hassine in the slack channel: https://filigran-community.slack.com/archives/CHZC2D38C/p1673599524232109

There are three related issues - I will raise a GH issue for two of them and then submit a proposed PR to fix both:
1 - Connector imports garbage data (HTML tags)
2 - Connector leaves trailing white-space at the end of Indicator/Observable name and value (Raised as #974)
3 - VXVault entities do not show up in bulk search, which is caused by issue 2, as bulk-search only does an exact match currently and can't match on the trailing `\n` characters. The developers are aware of this, so I will not raise an issue for it.


Discussion:

Resolving patch:
diff --git a/external-import/vxvault/src/vxvault.py b/external-import/vxvault/src/vxvault.py
--- a/external-import/vxvault/src/vxvault.py
+++ b/external-import/vxvault/src/vxvault.py
@@ -1,4 +1,5 @@
 import os
+import re
 import ssl
 import sys
 import time
@@ -7,6 +8,7 @@
 
 import certifi
 import yaml
+
 from pycti import OpenCTIConnectorHelper, get_config_variable
 from stix2 import TLP_WHITE, URL, Bundle, ExternalReference
 
@@ -101,6 +103,13 @@ def run(self):
                                 count += 1
                                 if count <= 3:
                                     continue
+                                line=line.strip()
+                                matchHtmlTag = re.search(r'^<\/?\w+>', line)
+                                if matchHtmlTag:
+                                    continue
+                                matchBlankLine = re.search(r'^\s*$', line)
+                                if matchBlankLine:
+                                    continue
                                 external_reference = ExternalReference(
                                     source_name="VX Vault",
                                     url="http://vxvault.net",


Test patch:
Repository: OpenCTI-Platform/connectors

Issue:
Error KeyError: 'value' in Shodan-InternetDB connector
## Description

We get the following error in Shodan-InternetDB connector for every IP we try to process:

INFO:root:Reading StixCyberObservable {f14b0557-269b-478c-822d-dd206ce88060}.
ERROR:root:Error in message processing, reporting error to API
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/pycti/connector/opencti_connector_helper.py", line 181, in _data_handler
    message = self.callback(json_data["event"])
  File "/opt/opencti/connectors/internal-enrichment/shodan-internetdb/src/shodan_internetdb/connector.py", line 103, in _process_message
    value = observable["value"]
KeyError: 'value'
INFO:root:Reporting work update_received work_6cbd1a73-9cfb-4825-9554-929cc42df702_2023-04-21T11:35:40.994Z
INFO:root:Message (delivery_tag=1) processed, thread terminated

## Environment

1. OS (where OpenCTI server runs): Ubuntu 22
2. OpenCTI version: 5.7.2
3. OpenCTI client: python



Discussion:

Resolving patch:
diff --git a/internal-enrichment/shodan-internetdb/src/shodan_internetdb/connector.py b/internal-enrichment/shodan-internetdb/src/shodan_internetdb/connector.py
--- a/internal-enrichment/shodan-internetdb/src/shodan_internetdb/connector.py
+++ b/internal-enrichment/shodan-internetdb/src/shodan_internetdb/connector.py
@@ -100,7 +100,7 @@ def _process_message(self, data: Dict[str, Any]) -> str:
                 return "Skipping observable (TLP)"
 
         # Process the observable value
-        value = observable["value"]
+        value = observable["observable_value"]
         if not validators.ipv4(value):
             log.error("Observable value is not an IPv4 address")
             return "Skipping observable (ipv4 validation)"
@@ -140,7 +140,7 @@ def format_list(alist: List[Union[str, int]]) -> str:
             """Format a list of primitives into a Markdown list"""
             return "".join(f"\n- {name}" for name in alist) or "n/a"
 
-        value = observable["value"]
+        value = observable["observable_value"]
         abstract = f"Shodan InternetDB enrichment of {value}"
         content = f"""```
 Shodan InternetDB:


Test patch:
Repository: oobabooga/text-generation-webui

Issue:
sd_api_pictures Should include SD metadata in generated images
I have the sd_api_pictures extension running and it seems like it is working as intended-
-The bot types something
-SD uses that as a prompt to generate an image
-The image appears in the chat, and also in the **/sd_api_pictures/outputs** directory.

**However, the output .PNG images do not have any Stable Diffusion metadata.**, which is very unfortunate.

_Originally posted by @altoiddealer in https://github.com/oobabooga/text-generation-webui/issues/309#issuecomment-1499245072_

<img width="945" alt="Screenshot 2023-04-08 092926" src="https://user-images.githubusercontent.com/1613484/230723811-8911a373-79a9-4655-9d96-877ef224193b.png">



Discussion:
This is how the generated images should include metadata (example from a typical SD output)

<img width="1258" alt="Screenshot 2023-04-08 093148" src="https://user-images.githubusercontent.com/1613484/230723931-3ecfb28c-5793-4738-a8d7-17d53ab67bc1.png">
I have modify the API, and it's work, the image has now metadata 

ln14 : `from PIL import Image, PngImagePlugin`

ln 147 : 
`png_payload = {
               "image": "data:image/png;base64," + img_str
            }  
            response2 = requests.post(url=f'{params["address"]}/sdapi/v1/png-info', json=png_payload)
            pnginfo = PngImagePlugin.PngInfo()
            pnginfo.add_text("parameters", response2.json().get("info"))`

corrected API 
[api.py.txt](https://github.com/oobabooga/text-generation-webui/files/11202380/api.py.txt)


Resolving patch:
diff --git a/extensions/sd_api_pictures/script.py b/extensions/sd_api_pictures/script.py
--- a/extensions/sd_api_pictures/script.py
+++ b/extensions/sd_api_pictures/script.py
@@ -138,14 +138,19 @@ def get_SD_pictures(description):
 
     visible_result = ""
     for img_str in r['images']:
-        image = Image.open(io.BytesIO(base64.b64decode(img_str.split(",", 1)[0])))
         if params['save_img']:
+            img_data = base64.b64decode(img_str)
+
             variadic = f'{date.today().strftime("%Y_%m_%d")}/{shared.character}_{int(time.time())}'
             output_file = Path(f'extensions/sd_api_pictures/outputs/{variadic}.png')
             output_file.parent.mkdir(parents=True, exist_ok=True)
-            image.save(output_file.as_posix())
+
+            with open(output_file.as_posix(), 'wb') as f:
+                f.write(img_data)
+
             visible_result = visible_result + f'<img src="/file/extensions/sd_api_pictures/outputs/{variadic}.png" alt="{description}" style="max-width: unset; max-height: unset;">\n'
         else:
+            image = Image.open(io.BytesIO(base64.b64decode(img_str.split(",", 1)[0])))
             # lower the resolution of received images for the chat, otherwise the log size gets out of control quickly with all the base64 values in visible history
             image.thumbnail((300, 300))
             buffered = io.BytesIO()


Test patch:
Repository: oobabooga/text-generation-webui

Issue:
extensions/openai KeyError: 'assistant'
### Describe the bug

Starting after [https://github.com/oobabooga/text-generation-webui/pull/2291]

Which I think it's a great improvement.

### Is there an existing issue for this?

- [X] I have searched the existing issues

### Reproduction

Start server with extension --openai --model openaccess-ai-collective_manticore-13b.
Starting [DGdev91 Auto-GPT](https://github.com/DGdev91/Auto-GPT), runs 1 cycle, give 'y' for the second, the error appears.

### Screenshot

_No response_

### Logs

```shell
openaccess-ai-collectiveException occurred during processing of request from ('127.0.0.1', 42032)
Traceback (most recent call last):
  File "/home/mihai/miniconda3/envs/textgen/lib/python3.10/socketserver.py", line 683, in process_request_thread
    self.finish_request(request, client_address)
  File "/home/mihai/miniconda3/envs/textgen/lib/python3.10/socketserver.py", line 360, in finish_request
    self.RequestHandlerClass(request, client_address, self)
  File "/home/mihai/miniconda3/envs/textgen/lib/python3.10/socketserver.py", line 747, in __init__
    self.handle()
  File "/home/mihai/miniconda3/envs/textgen/lib/python3.10/http/server.py", line 433, in handle
    self.handle_one_request()
  File "/home/mihai/miniconda3/envs/textgen/lib/python3.10/http/server.py", line 421, in handle_one_request
    method()
  File "/home/mihai/text-generation-webui/extensions/openai/script.py", line 310, in do_POST
    msg = role_formats[role].format(message=content)
KeyError: 'assistant'
----------------------------------------
```


### System Info

```shell
Win11 WSL2 Ubuntu 20.04
Python 3.10
```



Discussion:
This looks like 2 problems: 
1) you need to update your models/config.yaml (to include manticore) and/or your characters/instruction-following/Manticore Chat.yaml is missing.
2) There is a bug in the default template (says 'bot') should be 'assistant'

I will fix 2), for better results you should also fix 1).

Resolving patch:
diff --git a/extensions/openai/script.py b/extensions/openai/script.py
--- a/extensions/openai/script.py
+++ b/extensions/openai/script.py
@@ -259,7 +259,7 @@ def do_POST(self):
 
                 role_formats = {
                     'user': 'user: {message}\n',
-                    'bot': 'assistant: {message}\n',
+                    'assistant': 'assistant: {message}\n',
                     'system': '{message}',
                     'context': 'You are a helpful assistant. Answer as concisely as possible.',
                     'prompt': 'assistant:',


Test patch: