# Methodological Triangulation Using Neural Networks for Business Research

Authors: Steven Walczak
Journal: Advances in Artificial Neural Systems
 (2012)

---

## Abstract

Artificial neural network (ANN) modeling methods are becoming more widely used as both a research and application paradigm across a much wider variety of business, medical, engineering, and social science disciplines. The combination or triangulation of ANN methods with more traditional methods can facilitate the development of high-quality research models and also improve output performance for real world applications. Prior methodological triangulation that utilizes ANNs is reviewed and a new triangulation of ANNs with structural equation modeling and cluster analysis for predicting an individual's computer self-efficacy (CSE) is shown to empirically analyze the effect of methodological triangulation, at least for this specific information systems research case. A new construct, engagement, is identified as a necessary component of CSE models and the subsequent triangulated ANN models are able to achieve an 84% CSE group prediction accuracy.


---

## Body

## 1. Introduction

Artificial Neural networks (ANNs) have been used as a popular research and implementation paradigm in multiple domains for several decades now [1–9]. Recent literature is advocating the further usage of ANNs as a research methodology, especially in previously untried or underutilized domains [10, 11]. However, due to the early premise that ANNs are black boxes (i.e., it is difficult to evaluate the contribution of the independent variables) the demonstration of rigor and generalization of results from neural network research has been problematic.Similarities between ANNs and various statistical methods (which have been shown to be both rigorous and generalizable) have been described for potential adopters [10, 12]. A common research paradigm for ANN researchers is to compare results obtained using an ANN to other more traditional statistical methods, including regression [13–16], discriminant analysis [17–21], other statistical methods [22–24], and multiple statistical methods [25–28]. Of the 16 articles just referenced, the majority of these results show ANNs being either similar to (with 2 being similar) or better than (with 12 outperforming) the compared statistical methods within the specific application domain.While ANNs have a history, though short, their black box nature has led to adoption resistance by numerous-business related disciplines [29]. Methodological triangulation may help to overcome these adoption and usage reservations as well as providing a means for improving the overall efficacy of ANN applications. Methodological triangulation is the utilization of multiple methods on the same problem (empirical data) to gain confidence in the results obtained and to improve external validity [30, 31]. ANNs and traditional statistical methods are both quantitative in nature. A quantitative method is hereby defined as a specific tool, procedure, or technique that is used to analyze the data of a specific problem to produce a corresponding model or results to answer a business research question.The comparative analysis of ANNs versus standard statistical methods previously mentioned is an example of concurrent or parallel methodological triangulation [32], which is performed extensively to demonstrate performance improvements obtained through the utilization of neural network modeling. This paper will focus on nonconcurrent methodological triangulation techniques. Nonconcurrent methodological triangulation occurs when a statistical or other machine learning method is used in combination with an ANN, but the other method is applied to data prior to the ANN to refine the input vector and gain confidence in the reliability of the independent variables or alternately after the ANN has produced its results to improve the overall performance and/or interpretation of those results. The definition of nonconcurrent methodological triangulation used in this paper is similar to the sequential and parallel development mixed method and the sequential elaboration mixed method described by Petter and Gallivan [32].The research presented in this paper will assess the efficacy of utilizing nonconcurrent triangulation of method with ANNs, specifically the preselection of variables with recognized statistical techniques. The triangulated ANN will be applied to the case of estimating an individual’s computer self-efficacy (CSE) without relying on self-evaluation, since self-evaluation may be subject to numerous biases [33, 34]. The next section provides a brief background on methodological triangulation that has been previously applied with ANNs followed by a section that describes the CSE estimation problem in more detail, which serves as a classification problem to demonstrate the results of the new methodology. The fourth section will present the triangulation methodology and describe the developed ANN models for CSE estimation. The penultimate section presents the results and a discussion of these results for CSE estimation utilizing the triangulated ANN.


## 2. Background

This section describes ANNs and the literature on triangulation with ANNs.


### 2.1. Brief Description of ANNs

Before describing previous research that has either advocated or demonstrated the triangulation of various statistical and other machine learning methods with ANNs, a brief description of ANNs is provided. The following description is best suited to backpropagation trained ANNs, but can be generalized for other types of ANNs as well, especially other supervised learning ANNs. An ANN is a collection of processing elements typically arranged in layers (see Figure1). The input layer requires some type of numeric data value. These values are then multiplied by weights (another numeric value) and aggregated for each hidden layer processing element. Various aggregation functions may be used, but commonly either a standard summation or maximizing function is used, producing a value: hj=∑i=1nxiwi,j, which would be the aggregated input value for all input nodes (ranging over all possible i) for hidden processing node j. The hidden layer elements then transpose the aggregated input values using a nonlinear function, typically a sigmoid function, such that the output of each hidden node looks like: gj=(1+ek*Gain)-1. The outputs of each hidden layer node are then aggregated to the next layer, which may be the output layer or another hidden layer.Figure 1
Sample supervised learning ANN architecture (e.g., backpropagation).Learning, and hence the development of an accurate model, may be performed in a supervised or unsupervised manner. Supervised learning will be emphasized in this paper and utilizes historic examples of the problem being modeled. Examples of the independent variable sets are presented to the ANN, which then produces an output value or values as just described. The output value is compared to the known value from the historic training example and if an error above the error threshold exists, then the values of the weighted connections are adjusted to better approximate the observed output value. This type of learning is nonparametric and makes no assumptions about population distributions or the behavior of the error term [35].ANNs have been shown to be able to accurately approximate almost any type of model for both classification and forecasting (regression), including both linear and nonlinear models [36–38]. Evaluation of supervised learning ANN models is performed by withholding a portion of the historic data sets and using this as an out-of-sample verification of the generalization of the ANN solution model to the real world. When comparing NNs against other methods, it should be the error on these out-of-sample or other out-of-sample results that is compared, where out of sample implies data that was not used for development of the ANN model.


### 2.2. Previous Work with Triangulating NNs

As already mentioned in the previous section, comparison of ANN classification or forecasting results to standard statistical methods following a model selection approach [35, 39] is a common method used by researchers to attempt to demonstrate both the validity of using an ANN model and also to demonstrate a methodological improvement gained from the use of the ANN modeling paradigm. However, the focus of the research reported in this paper is on nonconcurrent method triangulation and as such this section will focus on previous research that has utilized statistical and other methods in a nonconcurrent manner with ANNs.A summary of prior research that implemented some form of triangulation that was not used for comparison of results is shown in Table1. Very early work in improving ANN architecture design employed genetic algorithms (GAs) or genetic programming (GP) prior to instantiation of the ANN. ANNs have also been triangulated concurrently with other ANNs in an ensemble to improve classification performance for problems where a single classifier cannot perform adequately. All other existing work utilizing triangulation occurs following the production of results by the ANN model to reduce output error from the ANNs or to improve the explanation of the output results.Table 1
Previous utilization of ANN triangulation.
Type of triangulationPurposeReferencesNC, priorFaster development and optimization of ANN architecture, utilizing GA and GP[40–44]NC, priorReduce dependent variable set[45]NC, afterReduction in output error using decision trees and parametric statistics[46–49]NC, afterImproved explanation of dependent variable effects[50–52]ConcurrentImprove classification for complex problems using ANN ensembles[53–55]NC, postIntegration of ensemble outputsregression [56]; decision tree [57]; heuristic [58]NC: nonconcurrent method triangulation.The proper selection of independent variables to utilize in the input vector for any ANN is required for performance optimization [45, 59–61]. Various techniques have been utilized in a nonconcurrent triangulation prior to training the ANN to eliminate correlated dependent variables and variables that have minimal or negative impact (noise) on the ANN results. These techniques include GAs to pre-select training data that will lead to faster convergence [43, 62], correlation matrices [60], regression [63], principal component analysis [29], and discriminant analysis [58].An interesting example of the power of preprocessing of possible input data comes from a study by Durand et al. [64]. They develop two models concurrently with the first being a partial least squares (PLS) regression model that has data preprocessed with a GA and an ANN model that has data preprocessed with a mutual information algorithm. The original problem space contained 480 variables. The GA was able to reduce the number of variables down to 11 for the PLS regression model and the mutual information algorithm was able to reduce the number of variables down to 12 for the ANN model, thus utilizing only 2.5 percent of the total available variables. The ANN model ultimately produced the best generalization performance between the two compared models.The preprocessing of input/training data or the postprocessing of ANN output data to improve its accuracy or understandability are advantageous techniques to improve ANN performance, at least within the limited domains where these techniques have been previously applied. A combination of both preprocessing and postprocessing is explored in the remainder of this paper for the case of classifying individual CSE.


## 2.1. Brief Description of ANNs

Before describing previous research that has either advocated or demonstrated the triangulation of various statistical and other machine learning methods with ANNs, a brief description of ANNs is provided. The following description is best suited to backpropagation trained ANNs, but can be generalized for other types of ANNs as well, especially other supervised learning ANNs. An ANN is a collection of processing elements typically arranged in layers (see Figure1). The input layer requires some type of numeric data value. These values are then multiplied by weights (another numeric value) and aggregated for each hidden layer processing element. Various aggregation functions may be used, but commonly either a standard summation or maximizing function is used, producing a value: hj=∑i=1nxiwi,j, which would be the aggregated input value for all input nodes (ranging over all possible i) for hidden processing node j. The hidden layer elements then transpose the aggregated input values using a nonlinear function, typically a sigmoid function, such that the output of each hidden node looks like: gj=(1+ek*Gain)-1. The outputs of each hidden layer node are then aggregated to the next layer, which may be the output layer or another hidden layer.Figure 1
Sample supervised learning ANN architecture (e.g., backpropagation).Learning, and hence the development of an accurate model, may be performed in a supervised or unsupervised manner. Supervised learning will be emphasized in this paper and utilizes historic examples of the problem being modeled. Examples of the independent variable sets are presented to the ANN, which then produces an output value or values as just described. The output value is compared to the known value from the historic training example and if an error above the error threshold exists, then the values of the weighted connections are adjusted to better approximate the observed output value. This type of learning is nonparametric and makes no assumptions about population distributions or the behavior of the error term [35].ANNs have been shown to be able to accurately approximate almost any type of model for both classification and forecasting (regression), including both linear and nonlinear models [36–38]. Evaluation of supervised learning ANN models is performed by withholding a portion of the historic data sets and using this as an out-of-sample verification of the generalization of the ANN solution model to the real world. When comparing NNs against other methods, it should be the error on these out-of-sample or other out-of-sample results that is compared, where out of sample implies data that was not used for development of the ANN model.


## 2.2. Previous Work with Triangulating NNs

As already mentioned in the previous section, comparison of ANN classification or forecasting results to standard statistical methods following a model selection approach [35, 39] is a common method used by researchers to attempt to demonstrate both the validity of using an ANN model and also to demonstrate a methodological improvement gained from the use of the ANN modeling paradigm. However, the focus of the research reported in this paper is on nonconcurrent method triangulation and as such this section will focus on previous research that has utilized statistical and other methods in a nonconcurrent manner with ANNs.A summary of prior research that implemented some form of triangulation that was not used for comparison of results is shown in Table1. Very early work in improving ANN architecture design employed genetic algorithms (GAs) or genetic programming (GP) prior to instantiation of the ANN. ANNs have also been triangulated concurrently with other ANNs in an ensemble to improve classification performance for problems where a single classifier cannot perform adequately. All other existing work utilizing triangulation occurs following the production of results by the ANN model to reduce output error from the ANNs or to improve the explanation of the output results.Table 1
Previous utilization of ANN triangulation.
Type of triangulationPurposeReferencesNC, priorFaster development and optimization of ANN architecture, utilizing GA and GP[40–44]NC, priorReduce dependent variable set[45]NC, afterReduction in output error using decision trees and parametric statistics[46–49]NC, afterImproved explanation of dependent variable effects[50–52]ConcurrentImprove classification for complex problems using ANN ensembles[53–55]NC, postIntegration of ensemble outputsregression [56]; decision tree [57]; heuristic [58]NC: nonconcurrent method triangulation.The proper selection of independent variables to utilize in the input vector for any ANN is required for performance optimization [45, 59–61]. Various techniques have been utilized in a nonconcurrent triangulation prior to training the ANN to eliminate correlated dependent variables and variables that have minimal or negative impact (noise) on the ANN results. These techniques include GAs to pre-select training data that will lead to faster convergence [43, 62], correlation matrices [60], regression [63], principal component analysis [29], and discriminant analysis [58].An interesting example of the power of preprocessing of possible input data comes from a study by Durand et al. [64]. They develop two models concurrently with the first being a partial least squares (PLS) regression model that has data preprocessed with a GA and an ANN model that has data preprocessed with a mutual information algorithm. The original problem space contained 480 variables. The GA was able to reduce the number of variables down to 11 for the PLS regression model and the mutual information algorithm was able to reduce the number of variables down to 12 for the ANN model, thus utilizing only 2.5 percent of the total available variables. The ANN model ultimately produced the best generalization performance between the two compared models.The preprocessing of input/training data or the postprocessing of ANN output data to improve its accuracy or understandability are advantageous techniques to improve ANN performance, at least within the limited domains where these techniques have been previously applied. A combination of both preprocessing and postprocessing is explored in the remainder of this paper for the case of classifying individual CSE.


## 3. Methodology for ANN Triangulation

As described in the Background section, method triangulation is already widely used for neural network research. However, the triangulation is normally mentioned in passing and the effect of the triangulation is not typically evaluated.One of the goals of the current research is to promote the ideal of utilizing method triangulation whenever ANNs are used in research or real world applications and to formalize to the extent possible a methodology for performing triangulation with ANNs. Another goal is to provide empirical evidence to demonstrate the efficacy of utilizing triangulation in ANN research.A flowchart for implementing triangulation is shown in Figure2. The methodology is focused on methodological triangulation that utilizes ANNs as one of 2 or more processes used nonconcurrently to develop robust research models or domain applications. The flowchart and proposed methodology does not include data preparation/cleansing, testing of multiple ANN architectures, or cross-comparison of different ANN learning methods; all of which are standard ANN model development practices [60, 61].Figure 2
Flowchart for method triangulation with ANNs.The alternate processes specified in the flowchart are meant to indicate that the researcher or developer has several choices here for which method to use in the triangulation process. Selection of a specific statistical method or other method is typically constrained by the from and qualities of the data to be analyzed.The proposed methodology emphasizes two significant issues with ANN development: improving models through noise reduction and improving the interpretation of results (to overcome the black-box nature of ANNs). A side benefit of the methods advocated for triangulation prior to the ANN is that any reduction in the independent variable set will reduce the overall costs of the model [13, 35].


## 4. The CSE Problem: A Case Study for Evaluating Methodological Triangulation with ANNs

To demonstrate the benefit of the proposed triangulation method paradigm, a case study to predict CSE using ANNs is shown. The technology acceptance model (TAM) introduced by Davis [65] has long been used to predict the adoption of new technology by users. CSE is strongly linked, even as a determinant, with the perceived ease of use component of the TAM model. Prior research on CSE is highlighted in Table 2.Table 2
Prior CSE research.
Purpose/findingsReferencesCSE shown to be determinant of PEU[52, 71–74]Definitions of CSE as ability to use computers[66, 75]Drior experience as antecedent of CSE[66, 68, 76–80]Computer anxiety as antecedent of CSE[66, 73, 75, 77, 81]Organizational support as antecedent of CSEorganization [74, 75, 78, 82] manager [83]Engagement as antecedent of CSE(e.g., playfulness or innovativeness)[73, 81, 84–87]In a technology environment, determining CSE is important because low CSE may hinder learning [66, 67], while participants scoring high in CSE perform significantly better in computer software mastery [68–70]. However, CSE is difficult to measure rigorously. Contradictory CSE results from prior research may be due to weaknesses in existing measures of the construct as well as the need for control of antecedent and consequent factors directly associated with CSE [66].Variables that measure the four constructs shown in Table2: prior experience; computer anxiety; organizational support; engagement, will make up the independent variable vector to the ANN.Although computer anxiety has similar measurement problems to CSE, prior experience and organizational support are easier to measure than CSE because they are more objective, can be validated, and are less dependent on perceptions. Even when perceptual measures are used for organizational support, as in this study, they are probably less emotionally laden than CSE and anxiety, which are tied to ego and self-assessment. Engagement, although a perceptual measure, is also less emotionally laden than CSE and computer anxiety and is observable. The nonparametric and nonlinear nature of ANNs make them ideal for modeling a problem that may have inaccurate or noisy data, such as in the evaluation of computer anxiety and engagement.This section introduced the CSE problem, which is solved using triangulation in the next section.


## 5. Triangulation to Improve ANN Performance

CSE variables to measure the four constructs: prior experience (PE), computer anxiety (CA), organizational support (OS), and engagement (E) are collected using a survey that is administered to undergraduate and graduate students at a large southwestern state university in the United States. Students in both undergraduate classes and graduate classes were given the survey over a four-semester period. A total of 239 surveys were returned. Questions for the survey were derived from previously validated research and are shown in the appendix, Table 7. Three responses were dropped because of incomplete information on prior experience yielding 236 fully completed surveys.The sample consisted of about 50% each of graduates and undergraduates and approximately two-thirds male. Almost 50% were of age 23 years or older. 93% had some work experience and 99% had more than one year of PC experience, thus minimizing differences in the prior experience variable values for the model.Although considered to be more of a data cleansing operation as opposed to triangulation to improve performance through noise elimination by reducing the variable data set, outlier analysis using the box plot method was performed [88]. Two responses were identified as outliers, thus producing a working data set of 234 responses.As shown in Table7 in the appendix, a total of 14 independent variables were collected to predict an individual’s CSE. As specified in the flowchart (see Figure 2), a correlation matrix of all independent variables was calculated [60] and high-correlation values indicated that all 4 of the prior experience variables were interdependent. Computer-based training and business application experience were dropped as having the highest correlation values. This is actually an interesting finding as a corollary result from the triangulation process, namely, that as prior work experience increases so does computer application experience. Eliminating the correlated variables reduced the independent variable set size from 14 variables down to 12 variables.The next triangulation step to occur prior to the implementation of the ANN model is dependent on both the type of problem being solved (e.g., prediction or classification) and the variable data. The student population is a subset of and meant to be demonstrative of what could be achieved with the ERP training tool population at large and thus the distribution of the general population in unknown. Many of the parametric statistical models require the assumption of a normally distributed population of answers.Structural equation modeling (SEM) is a methodology for analyzing latent variables, which cannot be measured directly. CSE and computer anxiety are examples of latent variables. However, SEM using LISREL, AMOS, and other similar packages makes assumptions of a normal distribution of the data and requires a relatively large sample size [89]. On the other hand, the partial least squares (PLS) method does not assume a normal distribution of the data and does not require as large a sample size as LISREL and similar statistical software packages. Therefore, PLS-based SEM serves as a triangulation statistical method to analyze the antecedent constructs for the CSE case study data.PLS-SEM is used to analyze a measurement model and a structural model. The measurement model determines whether the measures used are reliable and whether the discriminant validity is adequate. The loading on its construct assesses the reliability of an indicator. Loading values should be at least 0.60 and ideally at 0.70 or above indicating that each measure is accounting for 50% or more of the variance of the underlying latent variable [89]. Two indicators for CSE were dropped because loadings on the construct were lower than the threshold. This helps to reduce the overall cost of the subsequent model through reduction in the number of variables required.In Table3, composite reliability scores show high reliability for the final constructs. All values for composite reliability are greater than 0.8 (except prior experience which is measured with unrelated formative indicators that are not expected to highly correlate with each other). The diagonal of the correlation matrix shows the square root of the average variance extracted (AVE). The AVE square root values are greater than the correlations among the constructs supporting convergent and discriminant validity. The means and standard deviations of the construct scales are also listed.Table 3
Means, standard deviations, reliability, and correlations of constructs.
ConstructCSEOSPECAEngageμ4.504.753.363.664.24σ1.431.450.861.581.49CompositeReliability0.8530.8070.7170.8940.947CSE0.8120.317−0.1120.0700.550OS0.7630.0250.0050.415PE0.751−0.066−0.051CA0.8990.097Engage0/865Values on the correlations matrix diagonal (bold values) are square roots of the AVE.Table4 shows the loadings and cross-loadings to the constructs of the measures, which had adequate reliability. All loadings are greater (in an absolute value) than 0.7, and are greater (in an absolute value) than cross-loadings showing again strong convergent and discriminant validity.Table 4
Variable loadings and cross-loadings.
CSEOSPECAEng.CSE10.8060.217−0.1260.0580.421CSE20.8200.269−0.0420.0600.325CSE30.8100.241−0.0160.0590.353OS10.2120.779−0.055−0.0310.206OS20.2750.7450.092−0.0010.340OS30.1710.766−0.010−0.0240.223Work Experience−0.0940.0230.845−0.044−0.032PC Experience−0.0720.0130.644−0.059−0.050CA10.031−0.017−0.1270.818−0.010CA20.0580.127−0.0190.9740.171E10.5200.305−0.0610.0530.884E20.4830.335−0.1420.1290.841E30.4040.372−0.0180.1670.820E40.4920.404−0.0270.0430.883E50.3840.2790.0010.0290.855E60.4850.391−0.0270.0810.905In the SEM model, (see Figure3), the path coefficients produced by PLS show that engagement (0.492) and organizational support (0.112) are statistically significant (P<0.001). Prior experience (−0.091) and computer anxiety (0.016) are not statistically significant. Demographic variables for age and gender were also evaluated in the original SEM model, but neither was statistically significant (0.012 and 0.029, resp.) and are subsequently removed from the model. The antecedents for the model shown in Figure 3 explained 32% (R2) of the variance in CSE.Figure 3
Results from PLS-SEM analysis.From the PLS-SEM model, it appears that either the engagement (E) construct variables or the organizational support (OS) variables or perhaps the combination of these two constructs will produce the best performing ANN model to predict an individual’s CSE. As with most ANN research, various hidden node architectures are constructed for the three possible input vectors (E, OS, E, and OS combined) [35, 39]. Each ANN starts with a quantity of hidden nodes equal to half the number of input nodes and this value is incremented by one and the ANN retrained until the performance starts to decay, indicating overlearning. All architectures are trained utilizing the backpropagation (BP) learning algorithm and training is halted when a RMSE of less than 0.05 is reported by the training algorithm. Architectures with the number of hidden nodes equal to the number of input nodes almost universally outperformed the other architectures and samples of these architectures are shown in Figure 4.Figure 4
Backpropagation-trained ANN architectures for predicting CSE.Additional tests are performed evaluating ANNs trained using the radial basis function (RBF) training methodology, which should be more noise resistant and operates more efficiently in cases of extrapolation (versus interpolation) than BP [90]. As mentioned previously, comparing multiple ANN models and different supervised learning algorithms is a form of concurrent triangulation. The BP ANNs consistently outperformed the corresponding RBF ANNs and thus only the BP ANN results are reported.Since the research is interested in investigating the improvement to ANN performance through the utilization of statistical method triangulation with ANNs, other combinations of constructs are also evaluated to determine if theE, OS, or E and OS input vectors do indeed produce the optimal prediction performance. All combinations of E, OS, and computer anxiety (CA) are developed as ANN models and an additional ANN model that includes prior experience (X) is also implemented. Each of the different construct combinations ANNs follow the same multiple architecture development and training protocol as that used for the three different ANN models recommended by the PLS-SEM method. Each construct combination uniformly used all of the construct variables from the survey which were not eliminated by the correlation matrix, whenever that corresponding construct was part of the input vector.Additionally, an ANN model that utilized all 14 variables is developed and compared with the other results to further examine the benefit gained from triangulation. Post-ANN result triangulation to increase ANN model performance and understanding is discussed in the next section.


## 6. Results

In this section, the results of the pre-ANN model specification triangulation are examined and post-ANN triangulation to improve and explain ANN results are demonstrated.


### 6.1. PLS-SME Triangulation Identifies Optimal Input Constructs

The prediction performance for the best performing architecture for each of the various ANNs evaluated is displayed highlighted in Table5. Table 5 also shows the results of utilizing all variables, including the correlated variables eliminated earlier in the triangulation process. The evaluation is performed using a 12-fold cross-validation technique, which should approximate the obtainable results from utilizing a model that would have been trained on the full population set, but maintains the integrity of the data as all validation samples are withheld from the training data set for the 12 individual cross-validation ANN models [91]. Each ANN attempted to predict the composite CSE score, which was the summation of the three retained CSE variables (following the PLS analysis) and had a range from 3 to 21.Table 5
Artificial neural network CSE prediction result for 234 evaluation cases.
ConstructMAE of predict of CSEPerfect predict of CSEPredict within 1 of CSEE1.95620.09%50.85%OS2.34617.09%45.73%E, OS2.02214.96%49.15%CA2.59616.67%42.74%E, CA2.02917.95%51.28%OS, CA2.41915.81%44.87%E, OS, CA2.08813.68%47.86%E, OS, CA, X2.23516.67%41.03%Full model (all 14 var.)3.65010.26%21.37%[E: engagement, OS: organizational support, CA: computer anxiety, X: prior experience].As shown in Table5, the E only construct ANN, which was the most significant construct according to the PLS-SEM preprocessing, produced the smallest mean absolute error (MAE) term and also had the largest quantity of perfect predictions for predicting CSE. The combination of E and OS had the second smallest MAE. An additional column is presented in Table 5 that represents near misses for the ANN model CSE predictions. From the near-miss column, the E and CA combination model performs best on the near-miss evaluation, with the E only ANN model coming a close second (not statistically different) and the E and OS model in third place.Additionally, it can be seen that utilizing all 14 of the collected variables to predict the PLS-SEM modified CSE-dependent variable has a much worse performance than any of the reduced variable set. This provides strong empirical evidence for the need to triangulate using correlation matrixes to reduce variables and consequent noise [45, 60, 61].The question of how to evaluate this particular ANN prediction model arises from the various construct combinations shown in Table5. Based on the MAE and also correct predictions, the PLS-SME preprocessing was able to correctly identify the best set of independent variables for the ANN input vector. Exact matches of the CSE value may not be necessary and near misses (predictions within one of the actual value) may be just as useful. Interpreting the data in this way by recalculating the results based on an approximate match is a form of post-ANN method triangulation based on a heuristic, which is conceptually similar to cluster analysis. Expanding the analysis with this posttriangulation heuristic, shows that the addition of the CA construct variables enables the ANN to achieve optimal performance for predicting CSE within 1, though the E only ANN model was a very close second.Bansal et al. [13] make a strong case for simplifying ANN models to reduce cost and improve the interpretation and usability of the resulting model. Since the CA construct did not significantly improve the within one performance of the ANN, the reduced cost of the E only ANN may be sufficient to outweigh the small gains achieved through the addition of the CA construct. The utility of including the computer anxiety construct in an ANN CSE prediction model must be weighed against the cost of obtaining reliable measurements for this construct.The preceding example empirically demonstrates that utilizing traditional statistical methods can significantly improve ANN performance through the identification of the optimal set of independent variables, or in this case optimal antecedent constructs. The correlation matrix was able to eliminate 2 variables from the dependent variable input set. Due to the population and data constraints, the selected PLS-SEM triangulation was able to further reduce the input vector size and accurately identified the optimal constructs for inclusion in the ANN model to predict the exact CSE value from 3 to 21.


### 6.2. Post ANN Triangulation to Improve Results and Interpretation

As noted earlier, another utilization of methodological triangulation is to improve the performance or interpretation of ANN output. The results from Table5 already demonstrate that a posttriangulation heuristic method can improve results by 150 to 185 percent.For the case of predicting an individual’s CSE, an exact numeric value may not be necessary to utilize an ANN CSE prediction model’s output, since CSE is generally more broadly classified into levels or groups such as very high CSE, high CSE, moderate CSE, low CSE, and very low CSE. A further analysis of the BP-trained ANN CSE prediction model’s output is triangulated further to determine group identification and determine if this additional processing may improve the performance of the ANN CSE prediction models.Values for delineating different levels of CSE are applied to the three aggregated CSE variables from the survey to distinguish five different CSE levels for the population from very low to very high (very low, low, moderate, high, and very high). For example, an individual with a CSE score between 3 and 7 inclusive is placed in the very low CSE group. This may be performed statistically using ak-means clustering algorithm with the number of clusters set to 5. The predicted CSE output values are also converted into a group classification using the same cutoffs that are applied to the user responses. The new five-group category classification results are displayed in Table 6 for each of the ANN construct models reported previously in Table 5, with the highest prediction accuracy values for each column highlighted.Table 6
Group CSE classification triangulated from ANN output.
ConstructsPerfect   group predictionWithin-1-group predictionsPerfect fuzzified   group predictionE67.95%93.59%83.33%OS67.09%75.21%75.21%E, OS67.95%88.89%73.93%CA67.09%92.74%84.19%E, CA66.67%94.02%83.33%OS, CA67.95%89.74%76.92%E, OS, CA64.96%94.02%84.19%E, OS, CA, X61.54%94.02%79.19%Table 7
Survey instrument measures.
MeasureSurvey questionSourceCSEI feel more comfortable using the tool on my own with the tool’s Training Support Tool (TST)[83]CSEI would be able to use the tool with the TST even if there was no one around to show me how to use it.[75, 83]CSE1I expect to become very proficient with the tool by using the TST.[86]CSE2I feel confident that I can use the tool with the TST.[86]CSE3Using the TST probably helps me to be good with the tool[86]OS1The business school has provided most of the necessary help and resources to get us used to the TST quickly[74]OS2The business school realizes what benefits can be achieved with the use of the TST[74]OS3I am always supported and encouraged by my professor to use the TST in my course[74, 83]CA1I hesitate to use the tool without the TST for fear of making mistakes I cannot correct[74]CA2I feel apprehensive about using the tool without the TST[86]E1The TST keeps me totally absorbed in what I am doing.[85, 86]E2The TST holds my attention.[85]E3The TST is fun.[85]E4The TST is interesting.[85]E5The TST is engaging.[85]TheE-construct-only ANN CSE group classification prediction places the user in the correct CSE group 67.95% and within one group 93.59% of the time. The remaining predictions for all of the ANN CSE group classification models are within two groups of the correct classification (meaning that a very-low or low CSE user is never categorized as a very high CSE user and very high and high CSE users are never classified as very low CSE users). It should be noted that the traditional antecedents of CSE, CA and a combination of OS and CA also produce an identical perfect CSE group classification compared to the E-only construct ANN model.TheE-only ANN CSE prediction model with triangulated output did not produce the highest within-one-group predictions, similar to the case with the exact CSE value predictions, but had the second highest performance and again was not statistically different from the E and CA and also the E, OS and CA ANN models. As mentioned before, the additional cost associated with collected these variables versus the minimal performance increase needs to be evaluated to determine if the PLS-SEM constructs selection triangulation is merited.CSE level classifications of individuals may also be viewed as fuzzy sets [92, 93], with the specific cutoffs for placement within a group (i.e., the boundary conditions) being indeterminate, but contained. For example, an individual with an aggregated CSE score of 17 working in a typical setting may be considered to belong to the very high CSE group, but that same individual employed at IBM, Texas Instruments, or Lockheed Martin might only be placed in the high or even medium CSE group. An overlap of 1 to 2 points between groups is enabled to simulate the application of a triangulated fuzzy algorithm that transforms the results into a more compatible fuzzy set notation, meaning that boundary members may be viewed as belonging to two possible groups, but still maintaining high homogeneity of members within a group [92]. The fuzzy group classification results are also displayed in Table 6, again with the highest classification accuracy highlighted. The fuzzy classification results compared to the perfect nonfuzzy classification results indicate a 10 to 15 percent increase in classification accuracy for most of the ANN models and may ultimately be a more realistic approach for organizations in trying to determine the CSE level of employees.The fuzzy classification results displayed in Table6 lend further empirical support for inclusion of the E construct in the CSE prediction model, with the E-only model performing second highest and not statistically different from the highest prediction percentage. The E, OS, and CA fuzzy interpretation model and the CA-only fuzzy interpretations are the highest. This lends partial support for the earlier findings that E and possibly OS are the two most significant constructs. Another corollary finding is the potential for CA to be included as a required construct if optimal fuzzy model performance is desired. The PLS-SEM results displayed in Figure 3 show that CA is the only other construct to have a net positive impact on CSE, though this is fairly small.


## 6.1. PLS-SME Triangulation Identifies Optimal Input Constructs

The prediction performance for the best performing architecture for each of the various ANNs evaluated is displayed highlighted in Table5. Table 5 also shows the results of utilizing all variables, including the correlated variables eliminated earlier in the triangulation process. The evaluation is performed using a 12-fold cross-validation technique, which should approximate the obtainable results from utilizing a model that would have been trained on the full population set, but maintains the integrity of the data as all validation samples are withheld from the training data set for the 12 individual cross-validation ANN models [91]. Each ANN attempted to predict the composite CSE score, which was the summation of the three retained CSE variables (following the PLS analysis) and had a range from 3 to 21.Table 5
Artificial neural network CSE prediction result for 234 evaluation cases.
ConstructMAE of predict of CSEPerfect predict of CSEPredict within 1 of CSEE1.95620.09%50.85%OS2.34617.09%45.73%E, OS2.02214.96%49.15%CA2.59616.67%42.74%E, CA2.02917.95%51.28%OS, CA2.41915.81%44.87%E, OS, CA2.08813.68%47.86%E, OS, CA, X2.23516.67%41.03%Full model (all 14 var.)3.65010.26%21.37%[E: engagement, OS: organizational support, CA: computer anxiety, X: prior experience].As shown in Table5, the E only construct ANN, which was the most significant construct according to the PLS-SEM preprocessing, produced the smallest mean absolute error (MAE) term and also had the largest quantity of perfect predictions for predicting CSE. The combination of E and OS had the second smallest MAE. An additional column is presented in Table 5 that represents near misses for the ANN model CSE predictions. From the near-miss column, the E and CA combination model performs best on the near-miss evaluation, with the E only ANN model coming a close second (not statistically different) and the E and OS model in third place.Additionally, it can be seen that utilizing all 14 of the collected variables to predict the PLS-SEM modified CSE-dependent variable has a much worse performance than any of the reduced variable set. This provides strong empirical evidence for the need to triangulate using correlation matrixes to reduce variables and consequent noise [45, 60, 61].The question of how to evaluate this particular ANN prediction model arises from the various construct combinations shown in Table5. Based on the MAE and also correct predictions, the PLS-SME preprocessing was able to correctly identify the best set of independent variables for the ANN input vector. Exact matches of the CSE value may not be necessary and near misses (predictions within one of the actual value) may be just as useful. Interpreting the data in this way by recalculating the results based on an approximate match is a form of post-ANN method triangulation based on a heuristic, which is conceptually similar to cluster analysis. Expanding the analysis with this posttriangulation heuristic, shows that the addition of the CA construct variables enables the ANN to achieve optimal performance for predicting CSE within 1, though the E only ANN model was a very close second.Bansal et al. [13] make a strong case for simplifying ANN models to reduce cost and improve the interpretation and usability of the resulting model. Since the CA construct did not significantly improve the within one performance of the ANN, the reduced cost of the E only ANN may be sufficient to outweigh the small gains achieved through the addition of the CA construct. The utility of including the computer anxiety construct in an ANN CSE prediction model must be weighed against the cost of obtaining reliable measurements for this construct.The preceding example empirically demonstrates that utilizing traditional statistical methods can significantly improve ANN performance through the identification of the optimal set of independent variables, or in this case optimal antecedent constructs. The correlation matrix was able to eliminate 2 variables from the dependent variable input set. Due to the population and data constraints, the selected PLS-SEM triangulation was able to further reduce the input vector size and accurately identified the optimal constructs for inclusion in the ANN model to predict the exact CSE value from 3 to 21.


## 6.2. Post ANN Triangulation to Improve Results and Interpretation

As noted earlier, another utilization of methodological triangulation is to improve the performance or interpretation of ANN output. The results from Table5 already demonstrate that a posttriangulation heuristic method can improve results by 150 to 185 percent.For the case of predicting an individual’s CSE, an exact numeric value may not be necessary to utilize an ANN CSE prediction model’s output, since CSE is generally more broadly classified into levels or groups such as very high CSE, high CSE, moderate CSE, low CSE, and very low CSE. A further analysis of the BP-trained ANN CSE prediction model’s output is triangulated further to determine group identification and determine if this additional processing may improve the performance of the ANN CSE prediction models.Values for delineating different levels of CSE are applied to the three aggregated CSE variables from the survey to distinguish five different CSE levels for the population from very low to very high (very low, low, moderate, high, and very high). For example, an individual with a CSE score between 3 and 7 inclusive is placed in the very low CSE group. This may be performed statistically using ak-means clustering algorithm with the number of clusters set to 5. The predicted CSE output values are also converted into a group classification using the same cutoffs that are applied to the user responses. The new five-group category classification results are displayed in Table 6 for each of the ANN construct models reported previously in Table 5, with the highest prediction accuracy values for each column highlighted.Table 6
Group CSE classification triangulated from ANN output.
ConstructsPerfect   group predictionWithin-1-group predictionsPerfect fuzzified   group predictionE67.95%93.59%83.33%OS67.09%75.21%75.21%E, OS67.95%88.89%73.93%CA67.09%92.74%84.19%E, CA66.67%94.02%83.33%OS, CA67.95%89.74%76.92%E, OS, CA64.96%94.02%84.19%E, OS, CA, X61.54%94.02%79.19%Table 7
Survey instrument measures.
MeasureSurvey questionSourceCSEI feel more comfortable using the tool on my own with the tool’s Training Support Tool (TST)[83]CSEI would be able to use the tool with the TST even if there was no one around to show me how to use it.[75, 83]CSE1I expect to become very proficient with the tool by using the TST.[86]CSE2I feel confident that I can use the tool with the TST.[86]CSE3Using the TST probably helps me to be good with the tool[86]OS1The business school has provided most of the necessary help and resources to get us used to the TST quickly[74]OS2The business school realizes what benefits can be achieved with the use of the TST[74]OS3I am always supported and encouraged by my professor to use the TST in my course[74, 83]CA1I hesitate to use the tool without the TST for fear of making mistakes I cannot correct[74]CA2I feel apprehensive about using the tool without the TST[86]E1The TST keeps me totally absorbed in what I am doing.[85, 86]E2The TST holds my attention.[85]E3The TST is fun.[85]E4The TST is interesting.[85]E5The TST is engaging.[85]TheE-construct-only ANN CSE group classification prediction places the user in the correct CSE group 67.95% and within one group 93.59% of the time. The remaining predictions for all of the ANN CSE group classification models are within two groups of the correct classification (meaning that a very-low or low CSE user is never categorized as a very high CSE user and very high and high CSE users are never classified as very low CSE users). It should be noted that the traditional antecedents of CSE, CA and a combination of OS and CA also produce an identical perfect CSE group classification compared to the E-only construct ANN model.TheE-only ANN CSE prediction model with triangulated output did not produce the highest within-one-group predictions, similar to the case with the exact CSE value predictions, but had the second highest performance and again was not statistically different from the E and CA and also the E, OS and CA ANN models. As mentioned before, the additional cost associated with collected these variables versus the minimal performance increase needs to be evaluated to determine if the PLS-SEM constructs selection triangulation is merited.CSE level classifications of individuals may also be viewed as fuzzy sets [92, 93], with the specific cutoffs for placement within a group (i.e., the boundary conditions) being indeterminate, but contained. For example, an individual with an aggregated CSE score of 17 working in a typical setting may be considered to belong to the very high CSE group, but that same individual employed at IBM, Texas Instruments, or Lockheed Martin might only be placed in the high or even medium CSE group. An overlap of 1 to 2 points between groups is enabled to simulate the application of a triangulated fuzzy algorithm that transforms the results into a more compatible fuzzy set notation, meaning that boundary members may be viewed as belonging to two possible groups, but still maintaining high homogeneity of members within a group [92]. The fuzzy group classification results are also displayed in Table 6, again with the highest classification accuracy highlighted. The fuzzy classification results compared to the perfect nonfuzzy classification results indicate a 10 to 15 percent increase in classification accuracy for most of the ANN models and may ultimately be a more realistic approach for organizations in trying to determine the CSE level of employees.The fuzzy classification results displayed in Table6 lend further empirical support for inclusion of the E construct in the CSE prediction model, with the E-only model performing second highest and not statistically different from the highest prediction percentage. The E, OS, and CA fuzzy interpretation model and the CA-only fuzzy interpretations are the highest. This lends partial support for the earlier findings that E and possibly OS are the two most significant constructs. Another corollary finding is the potential for CA to be included as a required construct if optimal fuzzy model performance is desired. The PLS-SEM results displayed in Figure 3 show that CA is the only other construct to have a net positive impact on CSE, though this is fairly small.


## 7. Conclusion

This paper has presented evidence that triangulation of methods can improve the performance of ANN classification and prediction models. A case study of an ANN solution that predicts an individual’s CSE was reported to provide further empirical evidence of the efficacy in utilizing methodological triangulation when employing ANNs.The CSE prediction models utilized two different types of triangulation of methods: (1) preprocessing triangulation including both a correlation matrix and the statistical method (PLS_SEM) for sequential development [32] to identify the optimal set of independent variables for developing the ANN prediction model and (2) a post-ANN clustering heuristic and possibly an additional fuzzy set algorithm for sequential elaboration [32] to improve the classification performance of the ANN.The preprocessing triangulation effectively reduced the independent variable set and also succeeded in identifying the most relevant construct, engagement, orE. The E-construct-only BP ANNs achieved either optimal performance for perfect predictions of the exact value or group value for an individual’s CSE and had the lowest MAE. The triangulated clustering method may also help improve the understanding of the ANN’s output by transforming it into a more meaningful representation of the classification strategies that would be followed by a human resources department to identify computer application training needs of employees [94].The CSE prediction problem utilized to provide empirical results for the recommended triangulation methods is not a trivial case. Directly measuring CSE is problematic [66]. Utilizing antecedents to CSE that are measurable, such as engagement, provide reliable inputs to a CSE prediction model. This research has also demonstrated the need to include engagement in future research models of CSE. Subsequent output of a reliable CSE prediction may then be used as input to more complex models of technology acceptance and end-user training requirement models. Future research can investigate the use of ANN predicted CSE in the TAM and for more accurately predicting perceived ease of use.From the review of the literature and the results presented here, future research involving the development of ANN classification or prediction models should utilize appropriate statistical methods to determine input variables for the ANN model. Additionally, when appropriate, the demystification of ANN output through posttriangulation methods can only serve to improve adoption of ANN methodologies.While this paper has focused on how to triangulate statistical and other methods with ANN, where the ANN serves as the primary modeling paradigm, ANN themselves may also be utilized as a triangulating method to improve statistical and other modeling methods. Various researchers [95–97] have advocated the utilization of ANNs to assist in determining when a solution set has nonlinear properties and thus should not be modeled using strictly linear statistical modeling methods.Additionally, ANNs may be used to estimate posterior probabilities in classification problems [98]. Rustum and Adeloye [99] claim that ANNs may also be used to fill in missing data from domains that typically have very low quantities of data and where data may be noisy or missing, thus ANNs may be used to fill in the missing data reliably. This would then enable other research modeling techniques to be applied to larger and cleaner sets of data.The methodology proposed in this paper has been shown to be effective, at least for the domain of CSE and prior research has already utilized method triangulation, but without analyzing the effect of the triangulation. Some final precautions should be noted. The selection of both pre- and posttriangulation statistical tools and other methods to incorporate into any ANN research or development process is highly dependent on the type of data and goals of the research model. As noted in Figure3, alternate processes are available and selection of the appropriate statistical method is reliant on the type and constraints of the data. However, as demonstrated, if appropriate statistical and other methods (e.g., heuristic) are implemented, the results of the corresponding ANN models can be improved 750 percent or greater (difference between the full variable basin BP ANN model (bottom row of Table 5) and the perfect fuzzified predictions (Table 6)).



---

*Source: 517234-2012-03-08.xml*