Repository: socialwifi/RouterOS-api

Issue:
TypeError: can't concat str to bytes?
I got this `TypeError: can't concat str to bytes` idk why,i just trying simple code like this

```
import routeros_api as routers

connection = routers.RouterOsApiPool('IP', username='username', password='password')
api = connection.get_api()
api.get_binary_resource('/').call('ping', { 'address': '192.168.56.1', 'count': '4' }) #this ip i got from example
```

[This Example i Used](https://github.com/socialwifi/RouterOS-api/wiki/How-to-use)

And full traceback:
```
Traceback (most recent call last):
  File "mikro.py", line 5, in <module>
    api.get_binary_resource('/').call('ping', { 'address': '192.168.56.1', 'count': '4' })
  File "/home/sengkunibot/.local/lib/python3.7/site-packages/routeros_api/resource.py", line 39, in call
    additional_queries=additional_queries).get()
  File "/home/sengkunibot/.local/lib/python3.7/site-packages/routeros_api/resource.py", line 45, in call_async
    additional_queries=additional_queries)
  File "/home/sengkunibot/.local/lib/python3.7/site-packages/routeros_api/api_communicator/encoding_decorator.py", line 12, in call
    path, command, arguments, queries, additional_queries)
  File "/home/sengkunibot/.local/lib/python3.7/site-packages/routeros_api/api_communicator/async_decorator.py", line 6, in call
    tag = self.inner.send(*args, **kwargs)
  File "/home/sengkunibot/.local/lib/python3.7/site-packages/routeros_api/api_communicator/exception_decorator.py", line 11, in send
    return self.inner.send(*args, **kwargs)
  File "/home/sengkunibot/.local/lib/python3.7/site-packages/routeros_api/api_communicator/key_cleaner_decorator.py", line 11, in send
    queries=encoded_queries, additional_queries=additional_queries)
  File "/home/sengkunibot/.local/lib/python3.7/site-packages/routeros_api/api_communicator/base.py", line 19, in send
    self.send_command(command)
  File "/home/sengkunibot/.local/lib/python3.7/site-packages/routeros_api/api_communicator/base.py", line 37, in send_command
    self.base.send_sentence(command.get_api_format())
  File "/home/sengkunibot/.local/lib/python3.7/site-packages/routeros_api/sentence.py", line 59, in get_api_format
    formated.append(b'=' + key + b'=' + value)
TypeError: can't concat str to bytes
```

My server:
- Ubuntu 16.04.5 LTS (GNU/Linux 4.4.0-31-generic x86_64)
- Python3.7

Hints from issue comments:
@kramarz: I guess it should be get_resource instead of get_binary_resource. Can you confirm that?@Alnyz: > I guess it should be get_resource instead of get_binary_resource. Can you confirm that?

Idk, i just try [This Example](https://github.com/socialwifi/RouterOS-api/wiki/How-to-use#examples)
Maybe u can explain it@avloboda: I'm getting a similar error with `api = connection.get_api()`
I'm not able to actually form a connection.
`import routeros_api`
`connection = routeros_api.RouterOsApiPool('192.168.1.1', username='admin', password='password', plaintext_login=True)`
`api = connection.get_api()`

The below error appears after this last command. I am using Python3.4.
`Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib64/python3.4/site-packages/routeros_api/api.py", line 51, in get_api
    self.api.login(self.username, self.password, self.plaintext_login)
  File "/usr/lib64/python3.4/site-packages/routeros_api/api.py", line 82, in login
    response = self.get_binary_resource('/').call('login',{ 'name': login, 'password': password })
  File "/usr/lib64/python3.4/site-packages/routeros_api/resource.py", line 39, in call
    additional_queries=additional_queries).get()
  File "/usr/lib64/python3.4/site-packages/routeros_api/resource.py", line 45, in call_async
    additional_queries=additional_queries)
  File "/usr/lib64/python3.4/site-packages/routeros_api/api_communicator/encoding_decorator.py", line 12, in call
    path, command, arguments, queries, additional_queries)
  File "/usr/lib64/python3.4/site-packages/routeros_api/api_communicator/async_decorator.py", line 6, in call
    tag = self.inner.send(*args, **kwargs)
  File "/usr/lib64/python3.4/site-packages/routeros_api/api_communicator/exception_decorator.py", line 11, in send
    return self.inner.send(*args, **kwargs)
  File "/usr/lib64/python3.4/site-packages/routeros_api/api_communicator/key_cleaner_decorator.py", line 11, in send
    queries=encoded_queries, additional_queries=additional_queries)
  File "/usr/lib64/python3.4/site-packages/routeros_api/api_communicator/base.py", line 19, in send
    self.send_command(command)
  File "/usr/lib64/python3.4/site-packages/routeros_api/api_communicator/base.py", line 37, in send_command
    self.base.send_sentence(command.get_api_format())
  File "/usr/lib64/python3.4/site-packages/routeros_api/sentence.py", line 59, in get_api_format
    formated.append(b'=' + key + b'=' + value)
TypeError: can't concat bytes to str`

I'm not sure if I am missing something, but I followed the wiki guide to as well.@stevehaskew: Two parts to the answer:

Simple way to make it work is to change the fields to bytes, like this:

```
import routeros_api as routers

connection = routers.RouterOsApiPool('IP', username='username', password='password')
api = connection.get_api()
api.get_binary_resource('/').call('ping', { 'address': b'192.168.56.1', 'count': b'4' }) #this ip i got from example
```

However, it does raise the question about whether under Python3 that we should be handling better.@okazdal: I had the same issue.
I modified sentence.py file from library. Added lines with **. Seems to fix this but I don't know about any other side effects

    def get_api_format(self):
        formated = [self.path + self.command]
        for key, value in self.attributes.items():
            **if type(value) == str:**
                **value = str.encode(value)**
            formated.append(b'=' + key + b'=' + value)
        for query in self.queries:
            formated.extend(query.get_api_format())
        if self.tag is not None:
            formated.append(b'.tag=' + self.tag)
        return formated@davidc: You're using Python 3. Use Python 2 or see #47 47@sukawatd: I modified sentence.py file from library. It's work for me.
Check key or value type before concatenate.
if it string type convert it to byte type.
```
from six import string_types

def get_api_format(self):
    formated = [self.path + self.command]
    for key, value in self.attributes.items():
        if isinstance(key, string_types):
            key = key.encode()
        if isinstance(value, string_types):
            value = value.encode()
        formated.append(b'=' + key + b'=' + value)
    for query in self.queries:
        formated.extend(query.get_api_format())
    if self.tag is not None:
        formated.append(b'.tag=' + self.tag)
    return formated```@sereykol: > Two parts to the answer:
> 
> Simple way to make it work is to change the fields to bytes, like this:
> 
> ```
> import routeros_api as routers
> 
> connection = routers.RouterOsApiPool('IP', username='username', password='password')
> api = connection.get_api()
> api.get_binary_resource('/').call('ping', { 'address': b'192.168.56.1', 'count': b'4' }) #this ip i got from example
> ```
> 
> However, it does raise the question about whether under Python3 that we should be handling better.

Hi when I try writing the same it always load "routeros_api.exceptions.RouterOsApiConnectionError: [Errno 9] Bad file descriptor" What should I do?@AIT-AT: > I modified sentence.py file from library. It's work for me. Check key or value type before concatenate. if it string type convert it to byte type.
> 
> ```
> from six import string_types
> 
> def get_api_format(self):
>     formated = [self.path + self.command]
>     for key, value in self.attributes.items():
>         if isinstance(key, string_types):
>             key = key.encode()
>         if isinstance(value, string_types):
>             value = value.encode()
>         formated.append(b'=' + key + b'=' + value)
>     for query in self.queries:
>         formated.extend(query.get_api_format())
>     if self.tag is not None:
>         formated.append(b'.tag=' + self.tag)
>     return formated```
> ```

That works for me too. But I use "`str`" instead of "`string_types`".

Solution patch:
diff --git a/CHANGELOG.md b/CHANGELOG.md
index b80e6de..ba52286 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -4,13 +4,13 @@ Changelog for RouterOS-api
 ## 0.19.1 (unreleased)
 ----------------------
 
-- Nothing changed yet.
+- Fix decoding non utf-8 characters. Allow setting fallback encoding.
 
 
 ## 0.19.0 (2025-03-06)
 ----------------------
 
-- Add support for `!empty` reply word ([#103](https://github.com/socialwifi/RouterOS-api/pull/103))
+- Add support for `!empty` reply word ([#103](https://github.com/socialwifi/RouterOS-api/pull/103)).
 
 
 ## 0.18.1 (2024-05-28)
diff --git a/README.md b/README.md
index fc0c11c..2e2003b 100644
--- a/README.md
+++ b/README.md
@@ -15,15 +15,14 @@ Python API to RouterBoard devices produced by [MikroTik](https://mikrotik.com/)
 ### Connection
 
 ```python
-#!/usr/bin/python
-
 import routeros_api
 
-connection = routeros_api.RouterOsApiPool('IP', username='admin', password='')
+connection = routeros_api.RouterOsApiPool('IP', username='admin', password='', plaintext_login=True)
 api = connection.get_api()
 ```
+Use `plaintext_login=True` option when connecting to RouterOS version 6.43 and newer.
 
-#### Connect Options
+#### Connection parameters
 
 ```python
 routeros_api.RouterOsApiPool(
@@ -31,6 +30,7 @@ routeros_api.RouterOsApiPool(
     username='admin',
     password='',
     port=8728,
+    plaintext_login=True,
     use_ssl=False,
     ssl_verify=True,
     ssl_verify_hostname=True,
@@ -90,19 +90,51 @@ It is highly recommended only to use this option with SSL enabled.
 routeros_api.RouterOsApiPool(host, username='admin', password='', plaintext_login=True)
 ```
 
+#### Handling non UTF-8 characters
+
+The API does not assume any particular encoding, so non utf-8 characters will
+not be shown correctly.
+A default encoding can be specified if needed. For some cases the `latin-1` 
+encoding will be the best one to use, for others - `windows-1250` will work 
+better. 
+Ref. https://forum.mikrotik.com/viewtopic.php?t=106053#p528460
+
+It is possible to use specific encoding by defining custom default structure 
+like this:
+
+```python
+import collections
+import routeros_api
+from routeros_api.api_structure import StringField
+
+connection = routeros_api.RouterOsApiPool('ip', username='admin', password='password', plaintext_login=True)
+api = connection.get_api()
+# This part here is important:
+default_structure = collections.defaultdict(lambda: StringField(encoding='windows-1250'))
+api.get_resource('/system/identity', structure=default_structure).get()
+```
+
+
 ### Execute Commands
 
 Call this with a resource and parameters as name/value pairs.
 
 ```python
-api.get_binary_resource('/').call('<resource>',{ <dict of params> })
+api.get_resource('/').call('<resource>', { <dict of params> })
+```
+
+You can also use the "binary" version, but in this case, all dict values
+should be encoded to bytes and the result will be returned as bytes.
+
+```python
+api.get_binary_resource('/').call('<resource>', { <dict of params> })
 ```
 
 #### Examples
 
 ```python
-api.get_binary_resource('/').call('tool/fetch',{ 'url': "https://dummy.url" })
-api.get_binary_resource('/').call('ping', { 'address': '192.168.56.1', 'count': '4' })
+api.get_resource('/').call('tool/fetch', {'url': 'https://dummy.url'})
+api.get_resource('/').call('ping', {'address': '192.168.56.1', 'count': '4'})
 ```
 
 ### Fetch List/Resource
@@ -126,15 +158,15 @@ list_queues.get()
 ### Add rules
 
 ```python
-list.add(attribute="vale", attribute_n="value")
+list.add(attribute='vale', attribute_n='value')
 ```
 
-**NOTE**: Atributes with `-`, like `max-limit` use underscore `_`: `max_limit`
+**NOTE**: Attributes with `-`, like `max-limit` use underscore `_`: `max_limit`
 
 #### Example:
 
 ```python
-list_queues.add(name="001", max_limit="512k/4M", target="192.168.10.1/32")
+list_queues.add(name='001', max_limit='512k/4M', target='192.168.10.1/32')
 ```
 
 ### Update Values
@@ -146,7 +178,7 @@ list.set(id, attributes)
 #### Example:
 
 ```python
-list_queues.set(id="*2", name="jhon")
+list_queues.set(id='*2', name='john')
 ```
 
 ### Get element:
@@ -158,7 +190,7 @@ list.get(attribute=value)
 #### Example:
 
 ```python
-list_queues.get(name="jhon")
+list_queues.get(name='john')
 ```
 
 ### Remove element:
@@ -170,7 +202,7 @@ list.remove(id)
 #### Example:
 
 ```python
-list_queues.remove(id="*2")
+list_queues.remove(id='*2')
 ```
 
 ### Close conection:
@@ -181,25 +213,24 @@ connection.disconnect()
 
 ### Run script and get output
 
-The example script only prints "hello". Here's a simplifed example of how to run it and get the output:
+The example script only prints "hello". Here's a simplified example of how 
+to run it and get the output:
 
 ```
->>> api.get_resource("/system/script").get()[0]['source']
+>>> api.get_resource('/system/script').get()[0]['source']
 '/put "hello"'
->>> async_response = api.get_binary_resource('/').call('system/script/run', {"number": '0'.encode('utf-8')})
->>> async_response.__dict__
-{'command': <routeros_api.sentence.CommandSentence object at 0x73a0f2b3eba0>, 'done_message': {'ret': b'hello'}, 'done': True, 'error': None}
->>> async_response.done_message['ret']
-b'hello'
+>>> response = api.get_resource('/system/script').call('run', {'number': '0'})
+>>> response.done
+True
+>>> response.done_message['ret']
+'hello'
 ```
 
 ### Other Example:
 
 ```python
 list_address =  api.get_resource('/ip/firewall/address-list')
-list_address.add(address="192.168.0.1",comment="P1",list="10M")
-
-list_address.get(comment="P1")
-
-list_address.remove(id="*7")
+list_address.add(address='192.168.0.1', comment='P1', list='10M')
+response = list_address.get(comment='P1')
+list_address.remove(id=response[0]['id'])
 ```
diff --git a/routeros_api/api.py b/routeros_api/api.py
index 7ba3776..1cf1137 100644
--- a/routeros_api/api.py
+++ b/routeros_api/api.py
@@ -103,7 +103,8 @@ class RouterOsApi(object):
                 'login', {'name': login.encode(), 'response': hashed})
 
     def get_resource(self, path, structure=None):
-        structure = structure or api_structure.default_structure
+        if structure is None:
+            structure = api_structure.default_structure
         return resource.RouterOsResource(self.communicator, path, structure)
 
     def get_binary_resource(self, path):
diff --git a/routeros_api/api_communicator/encoding_decorator.py b/routeros_api/api_communicator/encoding_decorator.py
index e9dd1bf..ebbb953 100644
--- a/routeros_api/api_communicator/encoding_decorator.py
+++ b/routeros_api/api_communicator/encoding_decorator.py
@@ -1,3 +1,8 @@
+import logging
+
+logger = logging.getLogger(__name__)
+
+
 class EncodingApiCommunicator(object):
     def __init__(self, inner):
         self.inner = inner
@@ -16,6 +21,11 @@ class EncodingApiCommunicator(object):
 
     def transform_item(self, item):
         key, value = item
+        if not isinstance(value, bytes):
+            logger.warning(
+                'Non-bytes value passed as item value ({}). You should probably use api.get_resource() instead of '
+                'api.get_binary_resource() or encode arguments yourself.'.format(value))
+            value = value.encode()
         return (key.encode(), value)
 
     def decorate_promise(self, promise):
diff --git a/routeros_api/api_structure.py b/routeros_api/api_structure.py
index 7899956..307a7f8 100644
--- a/routeros_api/api_structure.py
+++ b/routeros_api/api_structure.py
@@ -24,11 +24,14 @@ class Field(object):
 
 
 class StringField(Field):
+    def __init__(self, encoding='utf-8'):
+        self.encoding = encoding
+
     def get_mikrotik_value(self, string):
-        return string.encode()
+        return string.encode(encoding=self.encoding, errors='backslashreplace')
 
     def get_python_value(self, bytes):
-        return bytes.decode()
+        return bytes.decode(encoding=self.encoding, errors='backslashreplace')
 
 
 class BytesField(Field):


Test patch:
diff --git a/tests/test_resource.py b/tests/test_resource.py
index 21d40c9..fe31b20 100644
--- a/tests/test_resource.py
+++ b/tests/test_resource.py
@@ -9,6 +9,7 @@ from routeros_api import api_structure as structure
 from routeros_api import resource
 from routeros_api.api_communicator import base
 
+STRING_STRUCTURE = {'string': structure.StringField()}
 BYTES_STRUCTURE = {'bytes': structure.BytesField()}
 BOOLEAN_STRUCTURE = {'boolean': structure.BooleanField()}
 
@@ -32,6 +33,40 @@ class TestTypedResource(unittest.TestCase):
             '/unknown/', 'set', arguments={'x': b'y'}, queries={},
             additional_queries=())
 
+    def test_string_resource_get(self):
+        communicator = mock.Mock()
+        response = base.AsynchronousResponse([{'string': b's'}], command='')
+        communicator.call.return_value.get.return_value = response
+        some_resource = resource.RouterOsResource(communicator, '/string', STRING_STRUCTURE)
+        result = some_resource.get()
+        self.assertEqual(result, [{'string': 's'}])
+
+    def test_string_resource_get_non_utf8_characters(self):
+        communicator = mock.Mock()
+        response = base.AsynchronousResponse([{'string': b'test-\xb9\xe6\xbf-test'}], command='')
+        communicator.call.return_value.get.return_value = response
+        some_resource = resource.RouterOsResource(communicator, '/string', STRING_STRUCTURE)
+        result = some_resource.get()
+        self.assertEqual(result, [{'string': 'test-\\xb9\\xe6\\xbf-test'}])
+
+    def test_string_resource_get_windows_1250_characters(self):
+        string_structure = {'string': structure.StringField(encoding='windows-1250')}
+        communicator = mock.Mock()
+        response = base.AsynchronousResponse([{'string': b'test-\xb9\xe6\xbf\x8f-test'}], command='')
+        communicator.call.return_value.get.return_value = response
+        some_resource = resource.RouterOsResource(communicator, '/string', string_structure)
+        result = some_resource.get()
+        self.assertEqual(result, [{'string': 'test-ąćżŹ-test'}])
+
+    def test_string_resource_get_latin_1_characters(self):
+        string_structure = {'string': structure.StringField(encoding='latin-1')}
+        communicator = mock.Mock()
+        response = base.AsynchronousResponse([{'string': b'hap ac\xb2'}], command='')
+        communicator.call.return_value.get.return_value = response
+        some_resource = resource.RouterOsResource(communicator, '/string', string_structure)
+        result = some_resource.get()
+        self.assertEqual(result, [{'string': 'hap ac²'}])
+
     def test_bytes_resource_get(self):
         communicator = mock.Mock()
         response = base.AsynchronousResponse([{'bytes': b'y'}], command='')