Behavioural App Testing
Objective:
To assess & validate the AI tool’s usability (Goal Generator & Writing Assistant AI), engagement & fairness by analysing user interactions & feedback across diverse demographic groups.

Data Used:
Total No. of Unique Profiles: 16
Total Data Points: 
Goal Generators: 134
Writing assistant: 430

			Writing Assistant	Goals Generator
Profile Parameters	Gender	Male	18.8	23.1
		Female	81.3	76.9
	Location	Singapore	31.3	38.5
		DIFC UAE	12.5	15.4
		China	18.8	23.1
		   India	12.5	15.4
		Hong Kong	6.3	7.7
		   Taiwan	18.8	0




Key Insights:
1.	No dependency of AI_Use_Case ( df=1 , x2= 9.12, p = .058)  on User_Rating: User ratings do not depend on the type of AI use case.
2.	There is no significance on user rating due to gender (df = 2, x2 = 1.63, F=1.42, PR =.02) Gender doesn’t significantly influence user rating, so no bias is there based on gender. 
3.	Significance on user rating due to band(df = 4, x2=15.48, F=13.50, PR =.00) & ethnicity(df= 4, x2=12.97, F=11.32, PR=.00): Variability in ratings due to band & ethnicity suggest a possible disparity in how different groups perceive AI tool. A deeper analysis is required.
4.	User ratings vary significantly across different job levels suggesting that users at different level may have varied expectations or experiences with AI functionality.
5.	There is no significant relationship between User Rating & Change Tone for writing assistant AI. The change tone feature doesn’t have a significant on user ratings, suggesting that tone adjustment may not be a primary factor in user satisfaction.
6.	There is a significant relationship between the User Rating & SF Action for writing assistant indicating that the functionality impacts user satisfaction.
7.	What this indicate is that users’ satisfaction of LLM responses are not influenced by the specific functionalities (i.e., goals generator or AI-assisted writing). While exact causes are required further research, possible reasons could be: the context of behavior testing to focus on content relevance and user satisfaction, and users potentially placing a stronger emphasis on the content from LLM over the interaction of how the results came about; lack of influence from demographic variables could also indicate that this may be a universal tendencies. The latter hypothesis could be an avenue for future works in human-centered AI

Observations:
Pros:
•	Tone adjustment capability:  Tool changes the tone of feedback by replacing confrontational language with more diplomatic & constructive wording.
Example:

Input                                                                   Output
You are far too aggressive in your style when talking to internal and, even worst, external stakeholders	Your communication style may benefit from a more diplomatic approach when interacting with both internal and external stakeholders.





•	It preserves the main points of the input.



Input                                                                   Output
You are a lousy - utterly lousy - performer and your leadership skills are pathetic.	I believe everyone has areas where they can improve, and I'm here to support you in any way I can.

•	It creates warning message against irrelevant content.
Example:
Input                                                                      Output
I can probably do these test cases in multi-lingual texts.  I just tested with a paragraph made up of Korean and English, and the tool produced an error. Quite interesting.  I'd want to know/test more if I had more time, but am afraid I have to stop here. 	It looks like something went wrong
We’re working on it. Please try again in a few minutes.


Cons:
•	Tool fails to demonstrate any warning message when the input provided is in harsh tone.
•	Tool’s output can sometimes be generic potentially lacking the specificity or personalization needed for meaningful feedback.
•	Tool could be less effective for high level users or leaders who need more sophisticated feedback or goal setting capabilities.
•	Handling sensitive feedback raises concerns.
•	Tool fails to capture the intention of the user.
Example:



Input                                                                      Output
I have no words to describe how badly you have performed this year. Did you achieve anything at all? I doubt it. Having you in the team is like trying to swim with a huge rock tied around my neck.	The performance this year has been subpar. Is there any notable achievement to speak of? It's uncertain. Your presence in the team feels burdensome, akin to navigating with a heavy weight dragging me down.














User Profile Distribution:

   



Techniques Used:
Text Evaluation Framework: Promptfoo, Spacy & NLTK
Approach: Evaluation of output based on relevance, content similarity, grammar & more. It checks if rephrased or paraphrased content retains original meaning. Based on scoring system which classifies the output as Pass above a certain threshold (>0.6).
Assessment: Pass 100%

User Rating: Based on a scale of 1 to 5:
  User Rating Distribution:


			Writing Assistant	Goals Generator	Overall
Profile Parameters	Gender	Male	3.8	4.13	3.98
		Female	4.23	4.21	4.22
	Location	Singapore	4.1	4.38	4.3
		DIFC UAE	4.38	4.00	4.19
		China	4.6	4.00	4.3
		   India	4.35	3.70	4.03
		Hong Kong	2.8	5.00	3.9
		   Taiwan	3.92	0.00	3.92
 	Overall	 	4.2	4.25	4.18

Hypothesis Testing:

Non Parametric tests
1. Chi Square test
To check the dependencies of variables. 

H0: There is no dependencies of user rating on AI Use Case.
H1: There is no dependencies of user rating on AI Use Case.
from scipy.stats import chi2_contingency
chitable = pd.crosstab(df.User_Rating,df.AI_Use_Case)
stats,p,dof,expected = chi2_contingency(chitable)
print(chitable)

Chitable

User_Rating                                          	AI_Use_Case  
	Goals Generator AI  	Writing Assistant AI
			
1	7	24	
2	0	25	
3	15	45	
4	25	98	
5	76	223	

print(stats,p)


9.12 0.058

From the above,  p value is >0.05 therefore Null hypothesis can be accepted.

Therefore there is No dependency of AI_Use_Case on User_Rating.
2. One way ANOVA: User Rating vs AI Use Case
H0: There is no significance on User Rating due to the AI Use Case
H1: There is a significance on User Rating due to the AI Use Case
model = ols('User_Rating~C(AI_Use_Case)',df).fit() # User Rating is continious & ~C for categorical, here :AI_Use_Case
oneway = sm.stats.anova_lm(model,type=2)   
                                                            #  type=2 for ANOVA & ANCOVA 

print(oneway)
 	df	sum_sq   	 mean_sq        	 F    	PR(>F)
C(AI_Use_Case)	1	3.434645	3.434645	2.566043	0.10977
Residual   	536	717.44	1.338499	  NaN       	NaN

Here Between variation is = 3.43 & within variation is = 717.43

Degree of freedom is 1 (2-1 for two category in AI_Use_Case) &

4397 (number of observation-number of categories (n- C))

Calculated F value= 2.57. A small F value implies that the differences in group means are comparable to the natural variation within the groups, making it difficult to conclude that the observed differences are statistically significant.

p value = 0.10977 which is >0.05 
Therefore the null hypothesis is accepted & Alternate is rejected. So there is no significance on User Rating due to the AI Use Case.
3. ANCOVA: significance on user rating due to gender, band & ethnicity

 	df	sum_sq   	 mean_sq        	 F    	PR(>F)
C(Gender)	2	3.256396	1.628198	1.420008	2.43E-01
C(Band)	4	61.923	15.48075	13.501298	1.75E-10
C(Ethnicity)	4	51.896379	12.974095	11.315157	8.15E-09
Residual   	526	603.117895	1.146612	  NaN       	NaN

There is no significance on user rating due to gender & significance due to band & ethnicity.

Goal Generator:

Total data point: 40
No. of User Profiles: 6
User Rating : 
Average: 3.9

   


  


 

Observations:
Female employees show more dissatisfaction compared to other gender.



Distribution: 
Assumptions:   Threshold for user satisfaction is considered as 3. 3 is considered as neutral.

Functionality	User Rating (In %)
	>3	>=3
Goal Generator	5	95




Writing Assistant:

Overall data collected : 144
User Rating Distribution:
  




For Change Tone:


 	User Rating (In %)	Average Rating (In Scale)
 	<3	>=3	
Change Tone Constructive	7	93	4.125
Change Tone: Personable	21	79	3.35
Change Tone: Professional	8	92	4.35

Hypothesis Testing:
A.	Find the correlation between User Rating & Change Tone
H0- There is not a significant relationship between User Rating & Change Tone
H1- There is a significant relationship between User Rating & Change Tone

stats,p = pearsonr(df_writing_assistant['User_Rating'],df_writing_assistant['AI Action'])
print(stats,p)
-0.0061 0.90

From the above values we can see that the 'p' value is > 0.05 so the null hypothesis is accepted
Therefore, there is no significant relationship between User Rating & Change Tone.

For Different SF Actions:
			Count of rating 1	Count of rating 2	Count of rating 3	Count of rating 4	Count of rating 5
Year End Self Assessment			6	7	15	25	42
Give Feedback			18	6	13	34	80
Year End Manager Assessment			0	5	11	35	34
Checkpoints (People Leader & Direct Report)		0	7	6	4	65


B.	User Rating v/s SF Action
H0: there is no significant relationship between the User Rating & SF Action.
H1: there is a significant relationship between the User Rating & SF Action.
df_writing_assistant['SF Action']=Label_encoder.fit_transform(df_writing_assistant['SF Action'])
stats,p = pearsonr(df_writing_assistant['User_Rating'],df_writing_assistant['SF Action'])

print(stats,p)
0.14 0.004

From the above values we can see that the 'p' value is < 0.05 so the null hypothesis is not accepted.
Therefore, there is a significant relationship between the User Rating & SF Action.
C.	Two way ANOVA: User Rating vs AI Action or SF Action
H0: There is no significance on Attrition due to the AI Action or SF Action
H1: There is a significance on Attrition due to the AI Action or SF Action	
model = ols('User_Rating~SF_Action+AI_Action',df_writing_assistant).fit() # Attrition is continious & ~C for categorical, here :BusinessTravel

twoway = sm.stats.anova_lm(model,type=2)   
                                                            #  type=2 for ANOVA & ANCOVA 

print(twoway)

 	df	sum_sq   	 mean_sq        	 F    	PR(>F)
SF_Action   	1	11.36639	11.366389	8.287405	0.0042
AI_Action 	1	0.008381	0.008381	0.006111	0.937729
Residual   	412	565.0686	1.371526	  NaN       	NaN

From above observation we can see that the p value for SF_Action is <0.05 therefore it has a significance on User_Rating, whereas, p value for AI_Action is >0.05 therefore it doesn't have impact on User_Rating.



Key Insights:

1.	No dependency of AI_Use_Case ( df=1 , x2= 9.12, p = .058)  on User_Rating: User ratings do not depend on the type of AI use case.
2.	There is no significance on user rating due to gender (n=, df = 2, x2 = 1.63, F=1.42, PR =.02) Gender doesn’t significantly influence user rating, so no bias is there based on gender. 
3.	Significance due to band(n=, df = 4, x2=15.48, F=13.50, PR =.00) & ethnicity(n = , df= 4, x2=12.97, F=11.32, PR=.00): Variability in ratings due to band & ethnicity suggest a possible disparity in how different groups perceive AI tool. A deeper analysis is required.
4.	User ratings vary significantly across different job levels suggesting that users at different level may have varied expectations or experiences with AI functionality.
5.	There is no significant relationship between User Rating & Change Tone for writing assistant AI. The change tone feature doesn’t have a significant on user ratings, suggesting that tone adjustment may not be a primary factor in user satisfaction.
6.	There is a significant relationship between the User Rating & SF Action for writing assistant indicating that the functionality impacts user satisfaction.

Summary:
•	No significant difference in user ratings by gender indicating gender neutral experience.
•	Significant differences in user ratings across job levels suggesting that users in different roles may have varied expectations or experiences.
•	Significant impact of ethnicity on user ratings which may reflect different perceptions or interactions across cultural groups.
•	Data shows that user mostly dissatisfied while using the functionality like Change Tone Personable (mostly for Give Feedback), 21% rating was under 3. But no significant relationship with user rating indicating that tone adjustment may not be a critical factor.
•	Statistical evidence shows that there is a dependency of user rating on different SF actions suggesting that this functionality is central to user satisfaction & interaction quality.






Mitigation Strategy:
1.These are early indicators of potential desirability signals of dissatisfaction -- we need to continue to monitor this through 
(A) value and impact stuides 
 (B) experience research with qualitative interviews for deeper feedback as to "why" this is happening
2. Continuous monitoring & feedback: Implement ongoing data collection to monitor the demographic impact.
3.Engage a diverse panel for regular to ensure the tool remains equitable across user groups.





Glossary:
1.Hyopothesis: A statement about a population about a population parameter that can be tested. Hypothesis are formulated as null & alternative statement.
2.Null hypothesis: A statement that there is no effect or no difference.
3.Alternative Hypothesis: A statement that contradicts the null hypothesis suggesting there is an effect or a difference.
4.p value: The probability of observing a test statistic at least as extreme as the one calculated, given the null hypothesis is true.A lower p value indicates stronger evidence against the null hypothesis.
5.Degree of freedom: The no. of independent values in a calculation that are free to vary.
6.Reject the null hypothesis: The decision made if the p value is greater than or equal to the significance level, suggesting that the sample provides enough evidence to conclude that the null hypothesis is false.
7.Fail to reject the null hypothesis: The decision made if the p value is greater than or equal to the significance level, suggesting that the sample doesn’t provide enough evidence to conclude that the null hypothesis is false.

Draft a white paper based on this analysis