A processor implemented method for structure-based drug design using a multi-modal deep learning model, comprising: processing, via one or more hardware processors, an input having a target protein for drug design by using: (a) a multi-modal deep learning model, wherein the multi-modal learning model comprises a graph attention-based variational auto-encoder (GAT-VAE) module, (b) a simplified molecular input line entry system based variational auto-encoder (SMILES-VAE) module, (c) a conditional molecular generator, and (d) a drug-target affinity predictor module; obtaining, via the one or more hardware processors, by using the GAT-VAE module from the target protein, a latent vector of at least one of active site graph comprising of key amino acid residues, wherein the GAT-VAE module is pretrained to learn structure and type of interactions from amino acids lining the active site residues of the target protein; obtaining, via the one or more hardware processors, by using the SMILES-VAE module, at least one latent vector from the target protein, wherein the SMILES-VAE module is pretrained to learn the grammar of small molecules; concatenating, via the one or more hardware processors, by using the conditional molecular generator, at least one latent vector of active site graph of the GAT-VAE module with the atleast one latent vector of the SMILES-VAE module to generate a set of molecules specific to the target protein; iteratively performing, via the one or more hardware processors, by a reinforcement learning framework on the concatenated latent vector to optimize at least one molecule by using the drug-target affinity predictor module to predict an affinity value for the set of molecules towards the target protein, wherein the DTA predictor module is pretrained using a drug protein dataset; and designing, via the one or more hardware processors, by using the conditional molecule generator, at least one optimized molecule with an affinity of the target protein is greater than a pre-defined threshold score. (a) a multi-modal deep learning model, wherein the multi-modal learning model comprises a graph attention-based variational auto-encoder (GAT-VAE) module, (b) a simplified molecular input line entry system based variational auto-encoder (SMILES-VAE) module, (c) a conditional molecular generator, and (d) a drug-target affinity predictor module; obtaining, via the one or more hardware processors, by using the GAT-VAE module from the target protein, a latent vector of at least one of active site graph comprising of key amino acid residues, wherein the GAT-VAE module is pretrained to learn structure and type of interactions from amino acids lining the active site residues of the target protein; obtaining, via the one or more hardware processors, by using the SMILES-VAE module, at least one latent vector from the target protein, wherein the SMILES-VAE module is pretrained to learn the grammar of small molecules; concatenating, via the one or more hardware processors, by using the conditional molecular generator, at least one latent vector of active site graph of the GAT-VAE module with the atleast one latent vector of the SMILES-VAE module to generate a set of molecules specific to the target protein; iteratively performing, via the one or more hardware processors, by a reinforcement learning framework on the concatenated latent vector to optimize at least one molecule by using the drug-target affinity predictor module to predict an affinity value for the set of molecules towards the target protein, wherein the DTA predictor module is pretrained using a drug protein dataset; and designing, via the one or more hardware processors, by using the conditional molecule generator, at least one optimized molecule with an affinity of the target protein is greater than a pre-defined threshold score.