Repository: pytorch/serve
Issue: Custom Requirements installation failed

Please have a look at [FAQ's](../../docs/FAQs.md) and [Troubleshooting guide](../../docs/Troubleshooting.md), your query may be already addressed.

Your issue may already be reported!
Please search on the [issue tracker](https://github.com/pytorch/serve/issues) before creating one.

## Context

We are using torchserve to deploy, predict and explain pytorch lightning models in kubelow.

For the inference service, we are using the official docker hub image - `pytorch/torchserve-kfs:0.4.0`

Since the docker image doen't contain `pytorch-lightning` package. We are added it in `requirements.txt` and bundling it using `torch-model-archiver`. 

When `install_py_dep_per_model` is set to `true` the packages listed in the `requirements.txt` are **not getting installed**. The issue is getting reproduced in both standalone and docker images.

* torchserve version: 0.4.0
* torch-model-archiver version: 0.4.0
* torch version: 1.8.1
* torchvision version [if any]: 0.9.1
* torchtext version [if any]: 0.9.1
* torchaudio version [if any]:
* java version: 
ubuntu@ubuntu-ThinkPad-L490:~/Documents/facebook/phase2/mlflow-torchserve/examples/IrisClassification$ java --version
openjdk 11.0.11-ea 2021-04-20
OpenJDK Runtime Environment (build 11.0.11-ea+4-Ubuntu-0ubuntu3.16.04.1)
OpenJDK 64-Bit Server VM (build 11.0.11-ea+4-Ubuntu-0ubuntu3.16.04.1, mixed mode, sharing)

* Operating System and version: Ubuntu - 16.04

## Your Environment
<!--- Include as many relevant details about the environment you experienced the bug in -->
* Installed using source? [yes/no]: no
* Are you planning to deploy it using docker container? [yes/no]: yes
* Is it a CPU or GPU environment?: CPU
* Using a default/custom handler? [If possible upload/share custom handler/model]: custom
* What kind of model is it e.g. vision, text, audio?: vision/text
* Are you planning to use local models from model-store or public url being used e.g. from S3 bucket etc.?
  [If public url then provide link.]: local model
* Provide config.properties, logs [ts.log] and parameters used for model registration/update APIs:
* Link to your project [if any]:

## Expected Behavior
When `install_py_dep_per_model` is set to `true`, pytorch lightning should be installed and model should load successfully.

## Current Behavior
when `install_py_dep_per_model` is set to `true`, torchserve is throwing the error (attached logs)

It is strange that, even if `pytorch_lightning` is present in the environment and `install_py_dep_per_model` is set to `true`, the same error is getting reproduced.

## Possible Solution
<!--- Not obligatory, but suggest a fix/reason for the bug -->

## Steps to Reproduce

[iris.zip](https://github.com/pytorch/serve/files/6677618/iris.zip)

1. Unzip iris.zip - It contains `iris.mar` and `config.properties` file
2. Create a directory `model-store`
3. copy iris.mar inside `model-store`
4. set `install_py_dep_per_model=false` in config.properties
5. Install pytorch lightning using pip - `pip install pytorch_lightning==1.3.2`
5. start torchserve with following command
`torchserve --start --ts-config config.properties`
6. Model will be loaded successfully.
7. Stop the torchserve - `torchserve --stop`
8. Uninstall pytorch lightining using `pip uninstall pytorch-lightning`
9. set `install_py_dep_per_model` to true now.
10. start torchserve with following command
`torchserve --start --ts-config config.properties`

## Failure Logs [if any]
Failure logs are attached in ts-failure.txt
[ts-failure.txt](https://github.com/pytorch/serve/files/6677666/ts-failure.txt)

Success scenario log
[ts-success.txt](https://github.com/pytorch/serve/files/6677678/ts-success.txt)




Changed files:
frontend/server/src/main/java/org/pytorch/serve/wlm/ModelManager.java

diff --git a/frontend/server/src/main/java/org/pytorch/serve/wlm/ModelManager.java b/frontend/server/src/main/java/org/pytorch/serve/wlm/ModelManager.java
index 372d2afd..b633b9e7 100644
--- a/frontend/server/src/main/java/org/pytorch/serve/wlm/ModelManager.java
+++ b/frontend/server/src/main/java/org/pytorch/serve/wlm/ModelManager.java
@@ -1,8 +1,10 @@
 package org.pytorch.serve.wlm;
 
 import com.google.gson.JsonObject;
+import java.io.BufferedReader;
 import java.io.File;
 import java.io.IOException;
+import java.io.InputStreamReader;
 import java.net.HttpURLConnection;
 import java.nio.file.Path;
 import java.nio.file.Paths;
@@ -145,10 +147,10 @@ public final class ModelManager {
             }
         }
 
-        logger.info("Model {} loaded.", tempModel.getModelName());
-
         setupModelDependencies(tempModel);
 
+        logger.info("Model {} loaded.", tempModel.getModelName());
+
         return archive;
     }
 
@@ -211,15 +213,40 @@ public final class ModelManager {
                             + requirementsFilePath; // NOPMD
 
             String[] envp =
-                    EnvironmentUtils.getEnvString(configManager.getModelServerHome(), null, null);
+                    EnvironmentUtils.getEnvString(
+                            configManager.getModelServerHome(),
+                            model.getModelDir().getAbsolutePath(),
+                            null);
+
             Process process =
                     Runtime.getRuntime()
                             .exec(
                                     packageInstallCommand,
                                     envp,
                                     model.getModelDir().getAbsoluteFile());
+
             int exitCode = process.waitFor();
+
             if (exitCode != 0) {
+
+                String line;
+                StringBuilder outputString = new StringBuilder();
+                // process's stdout is InputStream for caller process
+                BufferedReader brdr =
+                        new BufferedReader(new InputStreamReader(process.getInputStream()));
+                while ((line = brdr.readLine()) != null) {
+                    outputString.append(line);
+                }
+                StringBuilder errorString = new StringBuilder();
+                // process's stderr is ErrorStream for caller process
+                brdr = new BufferedReader(new InputStreamReader(process.getErrorStream()));
+                while ((line = brdr.readLine()) != null) {
+                    errorString.append(line);
+                }
+
+                logger.info("Dependency installation stdout:\\n" + outputString.toString());
+                logger.error("Dependency installation stderr:\\n" + errorString.toString());
+
                 throw new ModelException(
                         "Custom pip package installation failed for " + model.getModelName());
             }