A processor-implemented method for tuning graphics processing unit, GPU, parameters of GPU kernels, the method comprising: receiving a plurality of data regarding a GPU application, wherein the plurality of data regarding the GPU application includes a plurality of GPU parameters to be tuned for optimal functioning of the GPU kernels, the GPU application's architecture, a plurality of hardware counters and a plurality of performance counters, wherein the plurality of GPU parameters to be tuned include a thread per block parameter, a number of streams parameter, a global memory parameter and a L1/shared partition parameter and wherein the plurality of GPU parameters are utilized for performance tuning as execution on the GPU kernels, the GPU kernels being organized through a set of threads per block wherein the set of threads per block are mapped to the plurality of hardware counters on a set of streaming multiprocessors; representing a functional value for each of the plurality of GPU parameters as an optimization problem defined for in terms of a response function, wherein the response function includes at least one of the plurality hardware counters, wherein the functional value x * for each of the plurality of GPU parameters is represented as the optimization problem defined in terms of a response function f * ( x *) where * represents the GPU parameter and is defined as shown below: x ∗ = < x 1..... x n >, where opt f 1 x 1 ∀ x 1 ∈ P 1,..., opt f n x n ∀ x n ∈ P n and wherein the response function f 1 represents execution time of the GPU application under parameter configuration x 1 or profiling at least one of the plurality of hardware counters and wherein P 1,... P n represent the GPU parameters which are mutually exclusive; continuously monitoring and capturing behavior of the plurality of GPU parameters to identify at least one hardware counter from the plurality hardware counters that impacts the execution time of the GPU kernel and functioning of the plurality of GPU parameters to be tuned; dynamically tuning each of the plurality of GPU parameters to obtain the functional value of each of the plurality of GPU parameters for the optimization problem using at least one of a deterministic or a heuristic techniques using the plurality of hardware counters identified for each of the plurality of GPU parameters, wherein tuning each of the plurality of GPU parameters implements a set of heuristic techniques to decide course of the tuning for each of the plurality of GPU parameters based on the captured behavior of the plurality of hardware counters; and displaying each of the plurality of GPU parameters along with its dynamically tuned optimal functional value for optimal functioning of GPU kernels. receiving a plurality of data regarding a GPU application, wherein the plurality of data regarding the GPU application includes a plurality of GPU parameters to be tuned for optimal functioning of the GPU kernels, the GPU application's architecture, a plurality of hardware counters and a plurality of performance counters, wherein the plurality of GPU parameters to be tuned include a thread per block parameter, a number of streams parameter, a global memory parameter and a L1/shared partition parameter and wherein the plurality of GPU parameters are utilized for performance tuning as execution on the GPU kernels, the GPU kernels being organized through a set of threads per block wherein the set of threads per block are mapped to the plurality of hardware counters on a set of streaming multiprocessors; representing a functional value for each of the plurality of GPU parameters as an optimization problem defined for in terms of a response function, wherein the response function includes at least one of the plurality hardware counters, wherein the functional value x * for each of the plurality of GPU parameters is represented as the optimization problem defined in terms of a response function f * ( x *) where * represents the GPU parameter and is defined as shown below: x ∗ = < x 1..... x n >, where opt f 1 x 1 ∀ x 1 ∈ P 1,..., opt f n x n ∀ x n ∈ P n and wherein the response function f 1 represents execution time of the GPU application under parameter configuration x 1 or profiling at least one of the plurality of hardware counters and wherein P 1,... P n represent the GPU parameters which are mutually exclusive; continuously monitoring and capturing behavior of the plurality of GPU parameters to identify at least one hardware counter from the plurality hardware counters that impacts the execution time of the GPU kernel and functioning of the plurality of GPU parameters to be tuned; dynamically tuning each of the plurality of GPU parameters to obtain the functional value of each of the plurality of GPU parameters for the optimization problem using at least one of a deterministic or a heuristic techniques using the plurality of hardware counters identified for each of the plurality of GPU parameters, wherein tuning each of the plurality of GPU parameters implements a set of heuristic techniques to decide course of the tuning for each of the plurality of GPU parameters based on the captured behavior of the plurality of hardware counters; and displaying each of the plurality of GPU parameters along with its dynamically tuned optimal functional value for optimal functioning of GPU kernels.