Repository: triton-inference-server/server
Pull request: Fix memory leak caused by repeated calling Aws::InitAPI
Fixes issues outlined in #2794

Problem statement:
memory leak in s3 filesystem
**Description**
A clear and concise description of what the bug is.
There is a memory leak in s3 filesystem.
Here is the information given by valgrind
```
==627== 344 bytes in 1 blocks are definitely lost in loss record 1,637 of 2,513
==627==    at 0x483B7F3: malloc (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==627==    by 0x535D205: Aws::InitializeEnumOverflowContainer() (in /opt/tritonserver/lib/libtritonserver.so)
==627==    by 0x53880B3: Aws::InitAPI(Aws::SDKOptions const&) (in /opt/tritonserver/lib/libtritonserver.so)
==627==    by 0x4BFB5E9: nvidia::inferenceserver::(anonymous namespace)::GetFileSystem(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, nvidia::inferenceserver::(anonymous namespace)::FileSystem**) (filesystem.cc:1589)
==627==    by 0x4BFBFB3: nvidia::inferenceserver::IsDirectory(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool*) (filesystem.cc:1715)
==627==    by 0x4C7B0B5: nvidia::inferenceserver::ModelRepositoryManager::Create(nvidia::inferenceserver::InferenceServer*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::set<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, std::set<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, bool, std::unordered_map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::vector<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::vector<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > > > > const&, float, bool, bool, bool, double, std::unique_ptr<nvidia::inferenceserver::ModelRepositoryManager, std::default_delete<nvidia::inferenceserver::ModelRepositoryManager> >*) (model_repository_manager.cc:1017)
==627==    by 0x4D0CA52: nvidia::inferenceserver::InferenceServer::Init() (server.cc:178)
==627==    by 0x4D1CDA6: TRITONSERVER_ServerNew (tritonserver.cc:1537)
==627==    by 0x1A2DA4: main (main.cc:1192)
```
The memory leak caused by the `GetFileSystem` method in `filesystem.cc`, as it run `Aws::InitAPI(options);` every time but never use `Aws::ShutdownAPI(options)` to release it. The bug locates at the following code:
```
  // Check if this is an S3 path (s3://$BUCKET_NAME)
  if (!path.empty() && !path.rfind("s3://", 0)) {
#ifndef TRITON_ENABLE_S3
    return Status(
        Status::Code::INTERNAL,
        "s3:// file-system not supported. To enable, build with "
        "-DTRITON_ENABLE_S3=ON.");
#else
    Aws::SDKOptions options;
    Aws::InitAPI(options);
    static S3FileSystem s3_fs(options, path);
    *file_system = &s3_fs;
    return Status::Success;
#endif  // TRITON_ENABLE_S3
  }
```
It can be solved like this
```
  // Check if this is an S3 path (s3://$BUCKET_NAME)
  if (!path.empty() && !path.rfind("s3://", 0)) {
#ifndef TRITON_ENABLE_S3
    return Status(
        Status::Code::INTERNAL,
        "s3:// file-system not supported. To enable, build with "
        "-DTRITON_ENABLE_S3=ON.");
#else
    Aws::SDKOptions options;
    if(!S3FileSystem::GetS3Init()){
      Aws::InitAPI(options);
      S3FileSystem::SetS3Init(true);
    }
    static S3FileSystem s3_fs(options, path);
    *file_system = &s3_fs;
    return Status::Success;
#endif  // TRITON_ENABLE_S3
  }
```

**Triton Information**
What version of Triton are you using?
20.12

Are you using the Triton container or did you build it yourself?
Triton container

**To Reproduce**
Steps to reproduce the behavior.

Describe the models (framework, inputs, outputs), ideally include the model configuration file (if using an ensemble include the model configuration file for that as well).

**Expected behavior**
A clear and concise description of what you expected to happen.
No memory leak expected

Hints from issue discussion:
Thank you for brining this to our attention @yushcs. I have created a [PR](https://github.com/triton-inference-server/server/pull/2796) with the recommended changes here.

Commit message: Fix memory leak caused by repeated calling Aws::InitAPI

Patch to review:
diff --git a/src/core/filesystem.cc b/src/core/filesystem.cc
index 59a74786e3..9863f4f4bf 100644
--- a/src/core/filesystem.cc
+++ b/src/core/filesystem.cc
@@ -1753,7 +1753,8 @@ GetFileSystem(const std::string& path, FileSystem** file_system)
         "-DTRITON_ENABLE_S3=ON.");
 #else
     Aws::SDKOptions options;
-    Aws::InitAPI(options);
+    std::once_flag onceFlag;
+    std::call_once(onceFlag, [&options] { Aws::InitAPI(options); });
     static S3FileSystem s3_fs(options, path);
     *file_system = &s3_fs;
     return Status::Success;


Code review comments:
diff_hunk: @@ -1753,7 +1753,8 @@ GetFileSystem(const std::string& path, FileSystem** file_system)
         "-DTRITON_ENABLE_S3=ON.");
 #else
     Aws::SDKOptions options;
-    Aws::InitAPI(options);
+    std::once_flag onceFlag;
path: src/core/filesystem.cc
text: @user1:
This needs to be "static std::once_flag onceFlag;"